EXPRESS I'Smj lABEL NO. EI387347783DS 
Attorney Docket No. SONY-7471B ' 

H^TENT COOPERATION TREAli 



From the INT ERNATIONAL SEARCHING AUTHORITY 
To: 

LIMBACH & LIMBACH L.L.P. 
Attn . SUMMUT, Charles P. 
2001 Ferry Bui I ding ' 
SAN FRANCISCO, CALIFORNIA 94111 
UNITED STATES OF AMERICA 



RECEiVSOED 

AUG 1 01 ZOOM 
PCTLimbach Lanbtanhach 

NOTIFICATION OF TRANSMITTAL OF 
THE INTERNATIONAL SEARCH REPORT 
OR THE DECLARATION 

(PCT Rule 44.1) 



Date of mailing 
(day/month/year) 



08/08/2000 



Applicant's or agents file reference 

SONY-TOW ^Oi^'i '7^'7riL 



FOR FURTHER ACTION See paragraphs 1 and 4 Ijelow 



International application No. 

PCT/US 00/11504 



International filing date 
(day/month/ygar) 28/04/2000 



Applicant 

SONY CORPORATION et al 



1. [2 The applicant is hereby notified that the International Search Report has been estabUshed and is transmitted herewith. 
^l1^iSrs1:SS,rhen?"^^t^^^ C the mtema^onal Appiicatton (see Rule 46): 

When? The time limit for filing such amendments is nnrmally 2 months from the (tote of «'a'»"^J«?ifl*® , 
UrtlmTonrSearch Report; however, for more detAte. see the iStes on the accompanying sheet 

Whei«7 Directly to the International Bureau of WlPO 
34, chemin des Colombettes 
1211 Geneva 20, Switzerland 
Fasdmile No.: (41-22) 740.14.35 

For more detailed Instructions, see the notes on the accompanying sheet 

2 1-1 The applicant is hereby notified that no International Search Report will be established and that the declaration under 
■ I I Article 17(2)(a) to that effect is transmitted herewith. 

3 □ With regard to the protest against payment of (an) additional fee(s) under Rule 40.2, the applicant is notified that: 



□ 



no 



decision has been made yet on the protest; the applicant wUI be notified as soon as a decision is made. 



4 Further acUon(8): The applicant is reminded of the following: 

comptelion of the technical preparations for international publication. 

priority date or could not be elected because they are not bound by Chapter il. 



Name and mailing address of the International Searching Authority 
European Patent Office, P.B. 5818 Patentlaan 2 
NL-2280 HV Rijswijk 

Tel. (+31-70) 340-2040, Tx. 31 651 epo nl. 
Fax: (+31-70) 340-3016 



Authorized officer 

Shantlsaroop Pheral 



Form PCT/ISA/220 (July 1998) 



TO FORM PCT/ISA/220 



These ^k>te3 are intended to give the basic instructions concerning the filing of amendments under article 19. The 
Notes are based on the requirements of the Patent Cooperation Treaty, the Regulations and the Administrative Instructions 
urKtof that Treaty. In case of discrepancy between these Notes and those requirements, the latter are applicable. For more 
detailed information, see also the PCT Appltcant's Guide, a publication of WIPO. 

In these f^es, ^Article', "Rule", and "Section" refer to theprovisions of the PCT, the PCT Regulations and the PCT 
Administrative Instructions respectively. 



INSTRUCTIONS CONCERNING AMENDMENTS UNDER ARTICLE 19 

The applicant has, after having received the international search report, one opportunity to amend the claims of the 
international application. It should however be emphasized that, since all parts of the international application (claims, 
description and drawings) may bB amended during the international preliminary examination procedure, there is usually 
no need to file amendments of the claims under Article 1 9 except where, e.g. the applicant wants the latter to be published 
for the purposes of provisional protection or has another reason for amending the claims before international pbulication. 
Furthermore, it should be emphasized that provisional protection is available In some States only. 

What parts of the International application may be amended? 

Under Article 1 9, only the claims may be amended. 

During the international phase, the daims may also be amended (or further amended) under Article 34 before 
the International Preliminary Examirting Authority. The deschption ar>d drawings may only be amended under 
Article 34 before the Irttemational Examinirvg Authority. 

Upon entry into the national phase, all parts of the intemational application may be amended under Article 28 
or, where applieable. Article 41 . 

Whan? WitNn 2 months from the date of ti-ansmittal of the intemational search report or 1 6 months from the priority 

date, whict>«ver time linr^t expires later. It should l>e noted, however, that the amendmertts will be considered 
as having t>een rec^eived on time if they are received by the Intemational Bureau after the expiration of the 
applicablo time limit but tsefore the completion of the technical preparations for intemational publicalion 
(Rule 46.1). 

Wtiata not to flla tha amendmanta? 

The amerKiments may only be filed with the Intemational Bureau and not with ttte receiving Office or ttie 
Intemational SearoNrig Authority (Rule 46.2). 

Where a demand for intemational preliminary examination has beenAs filed, see below. 

How? Either by canoellirfg one or more entire daims, by adding one or more new claims or by amending the text of 

one or more of the claims as filed. 

A replacement sheet must be submitted for each sheet of the daims which, on account of an amendment or 
amendments, differs from the sheet originally filed. 

All the daims appearing on a replacement sheet must be numk»red in Arabic numerals. Where a daim is 
cancelled, no renumbering of the other claims is required. In all cases where daims are renumt^ered, they must 
be renumbered consecutively (Ac^inistrative Instructions, Section 205(b)). 

Tha amendments muat be made In the language In which the International application la to t»e pubilshad. 



What documanta must/may accompany tt>e amendments? 
Letter (Section 209(b)): 

Tha amendments must t>e sut>mitted with a letter. 

The letter will not be pufcilished with the intemational application and the amended daims. It should not be 
confused with the •Statement under Article 19(1)" (see below.under "Statement under Article 19(1)^. 

The letter must ba In English or French, at ttw choice of the applicant. However, If the language of the 
international application Is English, ttw letter must be In English; If the language of tha international application 
Is French, tha latter muat be In French. 



Motes to Form PCT/ISA/220 (first sheet) (January 1994) 

BNSOOCID: <XSISA220NOENP4J_> 



lOTES TO FORM PCT/ISA/220 (contlnOiH) 



The letter must indicate ttie differences between tfie claims as filed and the claims as amended. It must, in 
particular, indicate, in connection with each claim appearing in the intennational application (it being uriderstood 
that identical inctications concerning several claims may be grouped),whether 

(i) the claim is unchanged; 

00 the claim is cancelled; 

(iii) the claim is new; 

(rv) the claim replaces one or more claims as filed; 

(v) the claim is the result of the division of a claim as filed. 



The folloiwing examples Htustrate the mmnnBt in which amendments must be explained in the 
accompanying letter: 

1 . {Where originally there were 48 claims and after amendment of some claims there are 51 ]: 
"Claims 1 to 29, 31 , 32, 34, 35, 37 to 48 replaced by amended claims bearing ttie san>e numbers; 
datms 30, 33 and 36 unchanged; new daims 49 to 51 added.' 

2. [Where originally there were 1 5 claims and after amendment of all daims there are 1 1 ]: 
"Claims 1 to 1 5 replaced by amended daims 1 to 1 1 .* 

3. {Whcwe originally there were 1 4 daims and the amendments consist tn cancelling some daims and in adding 
new claims]: 

"Claims 1 lo 6 and 14 ur>changed; daims 7 to 1 3 canoeDed; new daims 1 5, 16 and 17 added." or 
"Claims 7 to 1 3 carolled; new claims 15,16 and 1 7 added; all other daims unchanged." 

4. p^ere various kinds of amendments are made]: 

"Claims 1-10 unchanged; dainns 11 to 13. 18 and 19canoeaed; claims 14, 15 and 16 replaced by amerwied 
daim 1 4; claim 1 7 subdivided into amended daims 15,16 and 1 7; new daims 20 and 21 added.* 



"Statement under article 19(1)" (Rule 46.4) 

The amendments may be aocompanied by a statement explaining the amendments arMj indicating any impad 
that such amendments might have on the dosehpbon and the drawings (which cannot be amended under 

Artide19(1)). 

The statement will bm published with the international application and the amended daims. 
It must be In tha language In which the international apppM ca tlon la to be publiahed. 

It must be brief, not exceeding 500 words if in English or if trartslated into English. 

ft should not be confused with and does not replaoe the letter indicating the differences between the claims 
as filed and as amended, ft must be filed on a separate sheet and nmjst t>e identified as such by a heading, 
preferably by using ttm words *Staten>ent urnier Article 1 9(1 ).* 

It may not contain any disparaging commerrts on the international search report or the relevance of citations 
contained in that report. Reference lo citations, relevant to a given daim, contair>ed in the international search 
report may tae made only in connection with an amendment of that claim. 



Consequence If a demaiMl for international preliminary examination has already l>een filed 

If, at ttie time of filing any amendments urxier Article 1 9, a demand for international preliminary examination 
has already been submitted, the applicant must preferably, at the same time of filing the amendments with the 
International Bureau, also file a copy of such amendments with the International Preliminary Examirung 
At^hority (see Rule 62.2(a), first sentence). 



Conaequencse with regard to translation of the Intematlonal application for entry Into the national phaae 

The applicant's attention is drawn to the fact that, where upon entry into the national phase, a translation of the 
daima as amended under Article 19 may have to tae furnished to the designated/elected Offices, instead of, or 
in addition to, the translation of the daims as filed. 



For further details on the requirements of each designated/elected Office, t 
Guide. 



> Volume II of the PCX Applicant's 



Notes to Fomn PCT/ISA/220 (second sheet) (January 1994) 

BNSOOCID: <XSlSA220NO€NP4j_> 



,^pATENT COOPERATION TREA 

PCT 



INTERNATIONAL SEARCH REPORT 

(PCT Articie 18 and Rules 43 and 44) 



Applicanf s or agent* s file reference 

SONY-T0747 


pQf) FURTHER Notification of Transmittal of International Searcfi Report 
. (Form PCT/ISA/220) as well as, wfiere applicable, item 5 t>elow. 

ACTION 


International application No. 

PCT/US 00/11504 


International filing date (day/montti/year) 

28/04/2000 


(Earliest) Priority Date (day/month/year) 

28/04/1999 


Applicant 

SONY CORPORATION et al . 



This International Search Report fias been prepared by this Intemational Searching Authority arKl is transmitted to the applicant 
according to Article 1 8. A copy is being transmitted to the International Bureau. 

This International Search Report consists of a total of 3 sfieets. 

|X| It is also accompanied by a copy of each prior art document cited in this report 



1 . Basis of the report 

a. With regard to the language, the international search was carried out on the basis of the intemational application in the 
language in which it was filed, unless otherwise indicated under this item. 

I I the intemational search was carried out on the t>asi8 of a translation of the intemational application furnished to this 
Authority (Rule 23. 1 (b)). 

b. With regard to any nucleotide and/or amino acid sequence disclosed in the intemational application, the international search 
was carried out on the basis of the sequence listing : 

I I contained in the intemational application in written form. 

filed together with tt^e international application in computer readat^le form. 



2. 
3. 



□ 
□ 
□ 
□ 

□ 

□ 
□ 



furnished sidssequentiy to this Authority in written form, 
furnished si^equentiy to this Authority in computer readt>le form. 

the statement that the subsequently furnished written sequence listing does not go t>eyorKl the disclosure in the 
international application as filed has t>een furnished. 

the statement that the information recorded in computer readable form is identical to the written sequence listing has been 
furnished 

Certain claims were found unsearchable (See Box I). 
Unity of Invention Is lacking (see Box It). 



With regard to the title, 

I I the text is approved as submitted by the applicant. 

[X] the text has t>een established by this Authority to read as follows: 

METHODS AND APPARATUS FOR COLOR DEVICE CHARACTERIZATION 



5. With regard to the abstract, 

|X| the text is approved as submitted by the applicant. 

□ the text has been established, according to Rule 3d.2(b). t>y this Authority as it appears in Box III. The applicant may, 
within one month from the date of mailing of this international search report, submit comments to this Authority. 

6. The figure of the drawings to be published with the abstract is Figure No. 5 

|X| as suggested by the applicant. None of the figures. 

|~| because the applicant failed to suggest a figure. 

I I t^ecause this figure betXeT characterizes the invention. 



Form PCT/ISA/210 (first sheet) (July 1998) 



INT ^NATIONAL SEARCH REPORT 

• 

A. CLASSinCATION OF SUBJECT MATTER , 

IPC 7 H04N9/64 H04N9/68 

According to International Patent Classification (IPC) or to both national dassification and IPC 

B. FIELDS SEARCHED 

A^ntmum documentation searched (classification system followed by classification symbols) 

IPC 7 H04N 



Documentation searched other than minimum documentation to the extent that such documents are included In the fields searched 



Electronic data base consulted during the international search (name of data base and, where practical, search terms used) 

PAJ, WPI Data, INSPEC, EPO-Internal 



latlonal Application Ho 

T/US 00/11504 



C. DOCUHEKTS CONSIDERED TO BE RELEVANT 



Category •* 


Citation of document, with incfication. whiere appropriate, of the relevant p>assage8 


Relevant to daim No. 


A 


EP 0 720 390 A (MATSUSHITA ELECTRIC IND CO 


1.4,10, 




LTD) 3 July 1996 (1996-07-03) 


19,24, 




27,33, 






47,50, 






65,70. 






73,79. 






88.95,96 




page 1, line 1 -page 2, line 19; figures 






1.2 






page 4, line 1 - line 52; figures 3-4B 






-/-- 





13 



Further documents are listed in the continuation of box C. 



10 



Patent family members are listed in annex. 



* Special categories of cited documents : 

'A' document defining the general state of the art which is not 
cor^dened to be of particular relevarx^ 

'E' earlier document but published on or after the international 
filing date 

"L" document wNch may throw doubts on priority claim(s) or 
which is dted to establish the putilication date of another 
citation or other special reason (as specified) 

'O* document referring to an oral disclosure, use, exfiibition or 
other means 

'P" docLmnent put)iished prior to the international filing date but 
later than the priority date daimed 



T" later document published after the international filir^ date 
or priority date arKi not in conflict with the application but 
cited to understand the principle or theory urxierlying the 
invention 

"X" document of particular relevance: the claimed invention 
cannot be considered novel or canrKt be considered to 
involve an inventive step wtien the document is taken alone 

"Y" document of particular relevance; the claimed invention 

canrwt be considered to invdve an inventive step when the 
document is combined with one or more other such docu- 
ments, such combtnation t>eing obvious to a person skilled 
in the art. 

document memt>er of ttie same patent family 



Date of the actual completion of the international search 

2 August 2000 


Date of mailing of the international search report 

08/08/2000 


Name ar>d mailing address of the ISA 

European Paterrt Office, P.B. 5818 Patentiaan 2 
NL - 2280 HV Rijswijk 
Tel. (401-70) 340-2040. Tx. 31 651 epo nl. 
Fax: (+31-70) 340-3016 


Authofized officer 

Fuchs, P 



Form PCT/ISAfilO (saoond sheet) (July 1992) 



page 1 of 2 



INI NATIONAL SEARCH REPORT 



national Application No 

fcT/US 00/11504 



C.(Contlnuation) DOCUMENTS CONSIDERED TO BE RELEVANT 



Category Citation of document, with indication, where appropriate, of the relevant passages 



Relevant to daim No. 



PATENT ABSTRACTS OF JAPAN 

vol. 016, no. 551 (P-1453), 

20 November 1992 (1992-11-20) 

& JP 04 208928 A (OLYMPUS OPTICAL CO LTD), 

30 July 1992 (1992-07-30) 



abstract 

US 5 376 963 A (ZORTEA ANTHONY E) 
27 December 1994 (1994-12-27) 



the whole document 



1.2,4,5, 

10,11, 

19,24, 

25,27, 

28,33, 

34,47, 

48,50, 

51,56, 

57,65, 

70.71, 

73,74. 

79,80, 

88,95.96 



1.4.10. 

19,24. 

27,33, 

47,50, 

65.70, 

73,79, 

88.95,96 



Form PCT/ISA^IO (continuation of second sheet) (Jufy 1992) 



page 2 of 2 



INT NATIONAL SEARCH REPORT 

ation on patent family members 



national Application No 

T/US 00/11504 



Patent document 




Publication 




Patent family 




Publication 


cited in search report 




date 




member(s) 




date 


EP 0720390 


A 


03-07-1996 


US 


5548697 


A 


20-08-1996 








DE 


69517401 


D 


13-07-2000 








JP 


8256349 


A 

n 


01-1 0-1QQfi 


.IP 0d9(\M9^ 


A 


"^n-n?— 1 QQ9 


NONE 






us 5376963 


A 


27-12-1994 


OE 


69424464 


D 


21-06-2000 








EP 


0618737 


A 


05-10-1994 








JP 


6295293 


A 


21-10-1994 








US 


5376962 


A 


27-12-1994 



Form PCT/ISA/210 (patent famity annex) (July 1992) 



m 



EXPRESS MAIL lABEL NO. EL387347783US 
Attorney Docket No. SaNY-747tB 

ENT COOPERATION TRE.i ~ 



From the INTERNATIONAL BUREAU 



PCT 

NOTIFICATION OF RECEIPT OF 
RECORD COPY 

(PCT Rule 24.2(a)) 



To: 



SUMMUT, r harla.; P 

Limbach & Limbach L.L.P. 
2001 Ferry Building 
San Francisco, CA 941 1 1 
ETATS-UNIS D'AMERIQUE 



•^tJi- 1 7 200Q 



-LP. 



Date of mailing (day/month/year) 

05 July 2000 (05.07.00) 



IMPORTANT NOTIFICATION 



International application No. ^pvxy^rfflJ^ [2^2^ 

PCT/USOO/11504 |o/^/0( ^ |(}^ 



Applicant's or agent* s file reference 



The applicant is hereby notified that the Internationaf Bureau has received the record copy of the international application as 
detailed below. 

Name(s) of the applies nt(s) and State(s) for which they are applicants: 

SONY CORPORATION (for all designated States except US) 

NISHIO, Kenichi et al (for US) 

(nternational filing date 28 April 2000 (28.04.00) 

Priority date{s) claimed 28 April 1999 (28.04.99) 
Date of receipt of the record copy 

by the International Bureau : 09 June 2000 (09.06.00) 

List of designated Offices : 

EP :AT,BE,CH,CY,DE,DK,ES,FI,FR,GB,GR,IEJT,LU,MC,NUPT,SE 
National :JP,US 



ATTENTION 

The applicant should carefully check the data appearing in this Notification. In case of any discrepancy between these data 
and the indications in the international application, the applicant should immediately inform the International Bureau. 

In addition, the applicant's attention is drawn to the information contained in the Annex, relating to: 



X 



time limits for entry into the national phase 
confirmation of precautionary designations 
requirements regarding priority documents 



A copy of this Notification is being sent to the receiving Office and to the International Searching Authority. 



The International Bureau of WlPO 
34, chemin des Colombettes 
121 1 Geneva 20, Switzerland 


Authorized officer: 

V. Gross 








Facsimile No. (41-22)740.14.35 


Telephone No. (41-22)338.83.38 








Form PCT/IB/301 (July 1998) \ 


003391149 



'iternationa I application No. 
ANNEX ^ORM PCT/IB/301 ^ PCT/USOQ/1 1 504 



INFORMATION ON TIME LIMITS FOR ENTERING THE NATIONAL PHASE 

The applicant is reminded that the "national phase" must be entered before each of the designated Offices indicated in the 
Notification of Receipt of Record Copy (Form PCT/lB/301) by paying national fees and furnishing translations, as prescribed by 
the applicable national laws. 

The time limit for performing these procedural acts is 20 MONTHS from the priority date or, for those designated States 
which the applicant elects in a demand for international preliminary examination or in a later election, 30 MONTHS from the 
priority date, provided that the election is made before the expiration of 1 9 months from the priority date. Some designated (or 
elected) Offices have fixed time limits which expire even later than 20 or 30 months from the priority date. In other Offices an 
extension of time or grace period, in some cases upon payment of an additional fee, is available. 

In addition to these procedural acts, the applicant may also have to comply with other special requirements applicable in 
certain Offices. It is the applicant's responsibility to ensure that the necessary steps to enter the national phase are taken in a 
timely fashion. Most designated Offices do not issue reminders to applicants in connection with the entry into the national 
phase. 

For detailed information about the procedural acts to be performed to enter the national phase before each designated 
Office, the applicable time limits and possible extensions of time or grace periods, and any other requirements, see the relevant 
Chapters of Volume II of the PCT Applicant's Guide. Information about the requirements for filing a demand for international 
preliminary examination is set out in Chapter IX of Volume I of the PCT Applicant's Guide. 

GR and ES became bound by PCT Chapter II on 7 September 1 996 and 6 September 1 997, respectively, and may, therefore, 
be elected in a demand or a later election filed on or after 7 September 1996 and 6 September 1997, respectively, regardless of 
the filing date of the international application. (See second paragraph above.) 

Note that only an applicant who is a national or resident of a PCT Contracting State which is bound by Chapter II has 
the right to file a demand for international preliminary examination. 

CONFIRMATION OF PRECAUTIONARY DESIGNATIONS 

This r.ctificaticn lists cn!y cpeciftc decignations made under Rule 4.9(3) in the request. It 's important to check that these 
designations are correct. Errors in designations can be corrected where precautionary designations have been made under 
Rule 4.9(b). The applicant is hereby reminded that any precautionary designations nnay be confirmed according to Rule 4.9(c) 
before the expiration of 1 5 months from the priority date. If it is not confirmed, it will automatically be regarded as withdrawn 
by the applicant. There will be no reminder and no invitation. Confirmation of a designation consists of the filing of a notice 
specifying the designated State concerned (with an indication of the kind of protection or treatment desired) and the payment 
of the designation and confirmation fees. Confirmation must reach the receiving Office within the 15-month time limit. 

REQUIREMENTS REGARDING PRIORITY DOCUMENTS 

For applicants who have not yet complied with the requirements regarding priority documents, the following is recalled. 

Where the priority of an earlier national, regional or international application is claimed, the applicant must submit a copy 
of the said earlier application, certified by the authority with which it was filed ("the priority document") to the receiving Office 
(which will transmit it to the International Bureau) or directly to the International Bureau, before the expiration of 1 6 months from 
the priority date, provided that any such priority document may still be submitted to the International Bureau before that date of 
international publication of the international application, in which case that document will be considered to have been received 
by the International Bureau on the last day of the 16-month time limit (Rule 17.1 (a)). 

Where the priority document is issued by the receiving Office, the applicant may, instead of submitting the priority 
document, request the receiving Office to prepare and transmit the priority document to the International Bureau. Such request 
must be made before the expiration of the 1 6-month time limit and may be subjected by the receiving Office to the payment 
of a fee (Rule 17.1(b)). 

If the priority document concerned is not submitted to the International Bureau or if the request to the receiving Office 
to prepare and transmit the priority document has not been made (and the corresponding fee, if any, paid) within the applicable 
time limit indicated under the preceding paragraphs, any designated State may disregard the priority claim, provided that no 
designated Office may disregard the priority claim concerned before giving the applicant an opportunity to furnish the priority 
document within a time limit which is reasonable under the circumstances. 

Where several priorities are claimed, the priority date to be considered for the purposes of computing the 16-month time 
limit is the filing date of the earliest application whose priority is claimed. 



Form PCT/IB/301 (Annex) (July 1998) 



003391149 



•r 



ENT COOPERATION TRE 



PCT/USOO/11504 



From the INTERNATIONAL BUREAU 



PCT 

NOTIFICATION CONCERNING 
SUBMISSION OR TRANSMITTAL 
OF PRIORITY DOCUMENT 

(PCT Administrative Instructions, Section 411) 


To: 

SAMMUT, Charles, P. 
LirriDacn a Lmnoacn l,L,P. 
2001 Ferry Building 
San Francisco, CA 941 1 1 
ETATS-UNIS D'AMERIQUE 


Date of mailing (day/month/year) 

10 October 2000 (10.10.00) 




Applicant's or agent's file reference 
SONY-747PC 


IMPORTANT NOTIFICATION 


International application No. 
PCT/US00/n504 y/^ 


International filing date (day/month/year) 
28 April 2000(28.04.00) ^ 


International publication date (day/nnonth/year) 

Not yet published 


Priority date (day/month/year) 

28 April 1999(28.04.99) ^ 


Applicant 

SONY CORPORATION et al 



1. The applicant is hereby notified of the date of receipt (except where the letters "NR" appear in the right-hand column) by the 
International Bureau of the priority document(s) relating to the earlier application(s) indicated below. Unless otherwise 
indicated by an asterisk appearing next to a date of receipt, or by the letters "NR", in the right-hand column, the priority 

uwVrftrfillWtik <j\^ilV/C'inVrVI woo oUtdTtt M^ls^V^ Of VI O I i wl I I) fclCV* kOtSiVf lIllCrltlOt.lv/iiCll ItH^I \iG lii t t ^ I » bl t i ■•■^ilitOiV^ t ^ , t yii f t \>^/' 

2. This updates and replaces any previously issued notification concerning submission or transmittal of priority documents. 

3. An asterisk!*) appearing next to a date of receipt, in the right-hand column, denotes a priority document submitted 
or transmitted to the International Bureau but not in compliance with Rule 17.1 (a) or (b). In such a case, the attention 
of the applicant is directed to Rule 17.1(c) which provides that no designated Office may disregard the priority claim 
concerned before giving the applicant an opportunity, upon entry into the national phase, to furnish the priority document 
within a time limit which is reasonable under the circumstances. 

4. The letters "NR" appearing in the right-hand column denote a priority document which was not received by the International 
Bureau or which the applicant did not request the receiving Office to prepare and transmit to the International Bureau, 

as provided by Rule 1 7.1 (a) or (b), respectively. In such a case, the attention of the applicant is directed to Rule 1 7.1 (c) which 
provides that no designated Office may disregard the priority claim concerned before giving the applicant an opportunity, 
upon entry into the national phase, to furnish the priority document within a time limit which is reasonable under the 
circumstances. 



Prioritv date 



28 Apri 1999 (28.04.99) 



Priority application No. 



11/122734 



Country or regional Office 
or PCT receiving Office 

JP 



Date of receipt 
of priority document 

26 Sept 2000 (26.09.00) 



The International Bureau of WlPO 


Authorized officer / \ 










34, chemin des Colombettes 




)h-Khuong 




1211 Geneva 20, Switzerland 






Facsihiile No. (41-22) 740.14.35 


Telephone No. (41-22) 33/83>^8 j 






Korm PCT/IB/304 (July 1998) 






003572075 



RECEIVED 

AUG 1 h 2000 

LIMBACH & LIMBACH i 



ENT COOPERATION TRt" 



PCT/US00/1150A 



From the INTERNATIONAL BUREAU 



PCT 



NOTIFICATION OF THE RECORDING 
OF A CHANGE 

(PCT Rule 92bis.1 and 
Administrative Instructions, Section 422) 



Date of mailing (day/month/year) 
02 August 2000 (02.08.00) 



Applicant's or agent's file reference 
SONY-747PC 



International application No. 
PCT/USOO/11504 



To: 



SAMMUT, Charles, P. 
Limbach & Limoach l. L..\ 
2001 Ferry Building 
San Francisco, CA 941 1 1 
ETATS-UNIS D'AMERIQUE 



IMPORTANT NOTIFICATION 



International filing date (day/month/year) 
28 April 2000 (28.04.00) 



Name and Address 


State of Nationality 


State of Residence 


SUMMUT, Charles, P. 








Telephone No. 






415 433 4150 






Facsimile No. 






415 433 8716 






Teleprinter No. 



1. The following indications appeared on record concerning: 
I I the applicant | | the inventor | X| the agent 



I I the common representative 



2. The International Bureau hereby notifies the applicant that the following change has been recorded concerning: 
I I the person 



the name [ | the address [ [ the nationality [ | the residence 



Name and Address 


State of Nationality 


State of Residence 


SAMMUT, Charles, P. 








Telephone No. 






415 433 4150 






Facsimile No. 






415 433 8716 






Teleprinter No. 



3. Further observations, if necessary: 

Please note that the agent's reference number has been changed from SONY-T0747 to 
SONY-747PC. 



4. A copy of this notification has been sent to: 
I X I the receiving Office 
I X I the International Searching Authority 
I I the International Preliminary Examining Authority 



[ I the designated Offices concerned 
I I the elected Offices concerned 
I I other: 



The International Bureau of WlPO 
34, chemin des Colombettes 
1211 Geneva 20, Switzerland 



Facsimile No.: (41-22) 740.14.35 



Authorized officer 



Christine Carrie 



Telephone No.: (41-22)338.83.38 



Form PCT/IB/306 (March 1994) 



003442492 



j^ENT COOPERATION TREJ^^ 



wo 00/65847 
PCT/USOO/11504 



From the INTERNATIONAL BUREAU 



PCT 



NOTICE INFORMING THE APPLICANT OF THE 
COMMUNICATION OF THE INTERNATIONAL 
APPLICATION TO THE DESIGNATED OFFICES 

(PCT Rule 47.1(c), first sentence) 



To: 



SAMMUT, Charles, P. 
Limbach 6t Limoacn l.L.P. 
2001 Ferry Building 
San Francisco, CA 941 1 1 
ETATS-UNIS D'AMERIQUE 



^OV 1 3 2000 



Date of mailing {day/month/yea r) 

02 November 2000 (02.1 1 .00) 


Limoacin Limbacr 




Applicant's or agent's file reference 
SONY-747PC 


IMPORTANT NOTICE 


International application No. 
PCT/USOO/11504 y 


International filing date (day/month/year) 
28 April 2000 (28.04,00) 


Priority date (day/month/year) 
28 April 1999 (28.04,99) 


Applicant 

SONY CORPORATION et al 



1. Notice is hereby given that the International Bureau has communicated, as provided in Article 20, the international application 
to the following designated Offices on the date indicated above as the date of mailing of this Notice: 

US 



In accordance with Rule 47.1(c), third sentence, those Offices will accept the present Notice as conclusive evidence that 
the communication of the international application has duly taken place on the date of mailing indicated above and no copy 
of the international application is required to be furnished by the applicant to the designated Office(s). 

2. The following designated Offices have waived the requirement for such a communication at this time: 
EPJP 



The communication wilt be made to those Offices only upon their request Furthermore, those Offices do not require the 
applicant to furnish a copy of the international application (Rule 49.1 (a-bis)). 

3. Enclosed with this Notice is a copy of the international application as published by the International Bureau on 
02 November 2000 (02.1 1.00) under No. WO 00/65847 



REMINDER REGARDING CHAPTER II (Article 31(2)(a) and Rule 54.2) 

If the applicant wishes to postpone entry into the national phase until 30 months (or later in some Offices) from the priority 
date, a demand for international preliminary examination must be filed with the competent International Preliminary 
Examining Authority before the expiration of 19 months from the priority date. 

It is the applicant's sole responsibility to monitor the 19-month time limit. 

Note that only an applicant who is a national or resident of a PCT Contracting State which is bound by Chapter II has the 
right to file a demand for international preliminary examination. 



REMINDER REGARDING ENTRY INTO THE NATIONAL PHASE (Article 22 or 39(1)) 

If the applicant wishes to proceed with the international application in the national phase, he must, within 20 months 
or 30 months, or later in some Offices, perform the acts referred to therein before each designated or elected Office. 

For further important information on the time limits and acts to be performed for entering the national phase, see the 
Annex to Form PCT/IB/301 (Notification of Receipt of Record Copy) and Volume II of the PCT Applicant's Guide. 



The International Bureau of WfPO 
34, chemin des Colombettes 
1 21 1 Geneva 20, Switzerland 



Facsimile No. (41-22) 740.14.35 



Authorized officer 

J. Zahra 

Telephone No. (41-22) 338.83.38 



Form PCT/IB/308 (July 1996) 



3613903 



HCI/USUU/ I lbU4 



REGEJVED 
ICT n 9 2000 



[TENT COOPERATION TRE^r' 



From the INTERNATIONAL BUREAU 



NOTIFICATION CONCERNING 
THE FILING OF AMENDMENTS OF THE CLAIMS 

(POT Administrative Instructions, Section 417) 


To: 

SAMMUT, Charles, P. 
Limbach & Linnbach L.L. P. 
2001 Ferry Building 
San Francisco, CA 941 1 1 
ETATS-UNIS D'AMERIQUE 


Date of mailing 

(day/month/year) 09 Contomhor onnn (OO HQ nn\ 
oeptemuer zuuu ^zz.uij.uu^ 


Applicant's or agent's file reference 
SONY-747PC 


IMPORTANT NOTIFICATION 


International application No. 

PCT/USOO/11504 


International filing date 

(day/month/year) 28 April 2000 (28.04.00) 


Applicant 

SONY CORPORATION et al 



1. The applicant is hereby notified that amendnnents to the clainns under Article 19 were received by the International Bureau on: 

18 September 2000 (18.09.00) 

2. This date is within the time limit under Rule 46.1. 

Consequently, the international publication of the international application will contain the amended claims according 
to Rule 48.2(f), (h) and (i). 



3. The applicant is reminded that the international application (description, claims and drawings) may be amended during 
the international preliminary examination under Chapter II, according to Article 34, and in any case, before each of the 
designated Offices, according to Article 28 and Rule 52, or before each of the elected Offices, according to Article 41 
and Rule 78. 





The International Bureau of WlPO 
34, chemin des Colombettes 
1211 Geneva 20, Switzerland 

Facsimile No.: (41-22)740.14.35 


Authorised officer . 

\ 

Dominique DELM>^ 

Telephone No.: (41-22)338.83.38 t 







EXPRESS MATT. lABEL NO. EL387347783US 
AttomeyJi?cket No. SONY-747US 



Patent cooperation treai 



^jDcket 



RECEiVSBED 



From the INTERNATIONAL SEARCHING AUTHORITY 



To: 



LIMBACH & LIMBACH L.L.P. 
Attn . SUMMUT, Charles P. 
2001 l-erry Bui I ding ' 
SAN FRANCISCO, CALIFORNIA 94111 
UNITED STATES OF AMERICA 



AUG 1 0 1 200(1000 
PCTLimbach 8i; Ltjntwaohach 



NOTIFICATION OF TRANSMITTAL OF 
THE INTERNATIONAL SEARCH REPORT 
OR THE DECLARATION 

(PCT Rule 44.1) 



Date of mailing 

(day/montlVyGar) 08/08/2000 



Applicant's or agent's file reference 



SeN¥-T©W 



FOR FURTHER ACTION See paragraphs 1 and 4 below 



International application No. 

PCT/US 00/ 11504 



International filing date 
(day/month/year) 28/04/2000 



Applicant 

SONY CORPORATION et al 



mm 



1 . 12 The applicant is hereby notified that the International Search Report has been established and is transmitted herewith. 

Flllna of amendments and Statement under Article 19: . . .. ^ /^o..i^^v. 

ThelSi^s he so wishes, to amend the claims of the International ApplicatK^n (see Rule 46). 

When? The time limit for filing such amendments is nomiallv 2 months from the date of transmittal of the 
klJem^onrsT^^^ Report; however, for more detSfe, see the ;jE»tes on the accompanying sheet. 



Where? Directly to the 



International Bureau of WlPO 
34, chemin des Colomt>eltes 
1211 Geneva 20, Switzerland 
Fascimite No.: (41-22) 740.14.35 



For more detailed Instructions, see the notes on the accompanying sheet. 



2. 



f-| The applicant is hereby notified that no International Search Report will be established and that the declaration under 
I — I Article 1 7(2)(a) to that effect is transmitted herewith. 



3. □ With regard to the protest against payment off (an) additional fee(s) under Rule 40.2. the applicant is notified that: 

i—l the Drotest toqether with the decision thereon has been transmitted to the International Bureau 

□ aw)K^ requ^tto forward the texts of both the protest and the decision thereon to the designated Offices. 

□ no decision has been made yet on the protest; the applicant will be notified as soon as a decision is made. 

4. Further actlon(s): The applicant is reminded of the following: 

qhortlv after 18 months from the priority date, the international application will be published by International Bureau 
5^ ao^^t^^ o avoid iTp^ a notice of withdrawal of the ^l^^^^^^'^^Z 

da^mu^^^^^ Bureau as provided in Rules 90bisA and 90b*s.3, respecUvely, before the 

completion of the technical preparations for international publication. 

priority date or could not be elected because they are not bound by Chapter II. 



Name and mailing address of the International Searching Authority 

European Patent Office. P.B. 5818 Patentlaan 2 

M\ NL-2280 HV Rijswijk 

Sff/ Tel. (+31 -70) 340-2040. Tx. 31 651 epo nl, 
Fax: (+31 -70) 340-301 6 



Authorized officer 

Shantisaroop Pherai 



Form PCT/ISA/220 (July 1998) 



NdWs TO FORM PCT/ISA/220 



These Notes are intended to grve the basic instaictions concerning the filing of amendments under article 19. The 
Notes are baaed on the requirements of the Patent Cooperation Treaty, the Regulations and the Admintstrative Instructions 
under that Treaty. In case of discrepancy between these Notes and those requirements, the latter are applicable. For more 
detailed information, see also the PCT Applicant's Guide, a publication of WIPO. 

In these Notes, "Article', "Rule", and 'Section" refer to the provisions of the PCT, the PCT Regulations and the PCT 
Administrative Instructions respectively. 



INSTRUCTIONS CONCERNING AMENDMENTS UNDER ARTICLE 19 



The applicant has, after having received the international search report, one opportunity to amend the claims of ttie 
international application. It should however be emphasized that, since all parts of the intemationaJ application (claims, 
description and drawings) may be amended during the irrtemational preliminary examination procedure, there is usually 
no need to file amendments of the claims urxier Article 19 except where, e g. the applicant wants the latter to be published 
for the purposes of provisional protection or has another reason for amendirig the claims t>efore irrtemational pbulication. 
Furthermore, it should t>e emphasized that provisional protection is available in some States only. 



What parts of the International application may t>e amended? 

Under Article 1 9, only the claims may be amended. 

During the international phase, the daima may also be amended (or further amended) under Article 34 before 
the Intemadional Preliminary Examining Authority. The description and drawings may only be amended under 
Article 34 before the International Excuninirfg Authority. 

Upon entry into the national phase, all parts of the intemattonal application may be amended under Article 28 
or, ¥tfhere applicable. Article 41 . 



When? WrtNn 2 months from the date of transmittal of the international search report or 1 6 nrtonths from the priority 

date, whichever time limit expires later. It should t>e noted, however, that the amendments will be oonsidered 
as having been received on time if they are received by the International Bureau after the expiration of the 
applicable time ttmtt but before the completion of the technical preparations for international puUicaiton 
(Rule 46.1). 



Where not to fUe the amendmenta? 

The amerxlments may only be filed with the Intemattonal Bureau and not with the receiving Office or the 
International Searching Authority (Rule 46.2). 

Where a demand for international preliminary examination has been^s filed, see below. 



How? Either by cancelling one or more entire claims, by adding one or more new claims or by amending the text of 

one or more of the claims as filed. 

A replacement sheet must be submitted for each sheet of the claims which, on account of an amendment or 
amendments, differs from the sheet originally filed. 

All the claims appearing on a replacement sheet must t>e numtiered in Arabic numerals. Where a daim is 
cancelled, no renumbering of the other claims is required. In all cases where dainrm are renumt»red, they must 
be renumlwred comecutively (Administrative Instructions, Section 205(b)). 

The amendmenta must be made In the language In which thelntematlonal application la to be publlshad. 



What documents must/may accompany tl«a amendmenta? 
Letter (Section 205(b)): 

The amendments must bm sutwnitted with a letter. 

The letter wiU not be published with the international application and the amended claims. It should not be 
confused with the 'Statement under Article 19(1)' (see below.under 'Statement under Article 19(1)*). 

The latter must be In English or French, at the choice of the applicant. However, If the language of the 
International application la English, ttw letter must be In English; if the language of the International appHcation 
Is Fretich, the latter must be In French. 



f 

Notes to Forni PCT/ISA/220 (first sheet) (January 1 994) 



BNSDOCIO: <XSISA220NOENP4J_> 



OTES TO FORM PCT/ISA/220 (continued) 



The fetter must indicate the differences between the claims as filed and the claims as amended. It must, in 
particular, indkate, in connection with each claim appearing in the international application (it being understood 
that idenbcal indications concerning several claims may be grouped), whether 

(i) the claim is unchanged; 

(ii) the claim is cancelled; 
(ill) the claim is new; 

(rv) the claim replaces one or more claims as filed; 

(v) the claim is the result of the division of a claim as filed. 



The follawing examples Illustrate the manner In which amendments must be explained In the 
accompanying letter: 

1 . [Where originally there were 48 claims and after amendment of some claims there are 51 ]: 
'Claims 1 to 29, 31 , 32, 34, 35, 37 to 46 replaced by amended claims tearing the same numbers; 
claims 30, 33 and 36 unchar>ged; new claims 49 to 51 added.* 

2. [Where originally there were 1 5 claims and after amendment of all claims there are 11]: 
"Claims 1 to 1 5 replaced by amended claims 1 to 1 1 

3. [Where origincUly there were 14 claims and the amendments consist in cancelHng some claims and in adding 
new claims]: 

"Claims 1 to 6 and 14 unchanged; claims 7 to 13 cancelled; new claims 15, 16 and 17 added.* or 
"Claims 7 to 1 3 cancelled; new claims 1 5, 16 and 1 7 added; all other claims uncharged.* 

4. [Where various kirKis of amendments are made]: 

'Claims 1*10 unchanged; claims 11 to 13, 18 and 19 cancelled; claims 14, 15 and 16 replaced by amerKied 
daim 14; claim 17 subdivided into amerxied claims 15, 16 and 17; new datms 20 and 21 added.* 



"^tatoment under wtlde 19(1)" (Rule 46.4) 

The amendments may tm accompanied t>y a statement explaining the amenc^ents arKi indicating arty impact 
that such amendments might have on the description and the cirawings (which caimot be amended under 
Article 19(1)). 

The statemerrt will be puNished with the irttemational appttcatton and the amended claims. 
II must be In the language in which the International apppUcation la to be puMlshed. 

It must be brief, not exceeding 500 words if in English or if translated into English. 

It should not be confused with and does not replace the letter ir>dicating the differences t)etween the claims 
as filed ar>d as amerKied. K must be filed on a separate sheet arxj must be identified as such by a heading, 
preferatsty by using the words 'Statement under Article 19(1).' 

It may not contain any disparaging comments on the international search report or the relevance of citations 
contained in that report. Reference to citations, relevant to a given claim, contair>ed in the intemational search 
report may be made only in connection with an amendment of that claim. 



Consec|uenoe if a demand for International preliminary examination has already l>een filed 

If, at the time of filing any amendments under Article 1 9, a demand for intemational preliminary examination 
has already been sut>mitted, the applicant must preferably, at the same time of filing the amendments with the 
Intemational Bureau, also file a copy of such amendments with the Intemational Preliminary Examining 
Authority (see Rule 62.2(a), first sentence). 



Consaquenoa with regard to translation of the intamallonal application for entry Into ttto national phaao 

The applicant's attention is drawn to the fact that, where upon entry into the national phase, a translation of the 
claims as amended under Article 19 may have to be furnished to the designated/elected Offices, instead of, or 
in addition to, the translation of the claims as filed. 

For further details on the requirements of each designated/elected Office, see Volume II of the PCT ApplicOTt's 
Guide. 



Notes to Form PCT/ISA/220 (second sheet) (January 1994) 

BNSOOCID: <XS1SA220NOENP4J_> 



^^ATENT COOPERATION TREAl^ 

PCT 



INTERNATIONAL SEARCH REPORT 



(PCT Article 18 and Rules 43 and 44) 


Applicants or agenfs file reference 

SONY-T0747 


FOR FURTHER Notificatton of Transmittal of International Search Report 
, _ (Form PCT/ISA/220) as well as, where applicable, item 5 below. 

ACTION 


International application No. 


International filing date (day/month/y&ar) 


(Earliest) Priority Date (day/month/year) 


PCT/ us 00/11504 


28/04/2000 


28/04/1999 


Applicant 






SONY CORPORATION et al . 







This International Search Report has been prepared by this International Searching Authority and is transmitted to the applicant 
according to Article 1 8. A copy is being transmitted to the International Bureau. 



This International Search Report consists of a total of 3 sheets. 

fX| It is also accompanied by a copy of each prior art document cited in this report 



Basis of the report 

a. With regard to the language, the international search was carried out on the basts of the international application in the 
language in which it was filed, unless ottierwise indicated under this item. 

I I the intemational search was carried out on the basis of a translation of the international application furnished to this 
Authority (Rule 23.1 (b)). 

b. With regard to any nucleotide and/or amino acid aequenoe disclosed in the intemational application, the international search 
was carried out on the basis of the sequence listing : 

I I contained in the international application in written form. 

I I filed together with the international application in computer readable form. 

I I fumished subsequently to this Authority in written form. 

I I fumished sid^sequently to this Authority in computer readtste form. 

I I the statement that the sut>sequently fumished written sequence listing does not go t>eyond the disclosure in the 
international application as filed has been fumished. 

I I the statement that the information recorded in computer readable form is identical to the written sequence listing has t>een 
furnished 



2. Certain claims were fourKi unsearchable (See Box I). 

3. Q Unity of Invention Is lacking (see Box tl). 



4. With regard to the title, 

I I the text is approved as submitted by the applicartt. 

|T] the text has been established by this Authority to read as follows: 

METHODS AND APPARATUS FOR COLOR DEVICE CHARACTERIZATION 



5. With regard to the abstract, 

|X| the text is approved as submitted by the applicant. 

□ the text has been established, according to Rule 38.2(b), by this Authority as it appears in Box III. The applicant may, 
within one month from the date of mailing of this international search report, submit comments to this Authority. 

6. The figure of the drawings to be put>lished with the abstract is Figure No. 5 



[X] as suggested by the applicant. None of the figures. 

I I because the applicant failed to suggest a figure. 

I I t)ecause this figure tetter characterizes the invention. 



Fomi PCT/ISA/210 (first sheet) (July 1998) 



PC# 



REQUEST 



The undersigned requests that the present 

international application be processed 
according to the Patent Cooperation Treaty. 



EXPRESS IS?^ LABEL NO, EL387347783U5 
Attome^^ckBt No. SONy-747US 

pl^Hliving Office use only i 



International Application No. 



International Filing Date 



Name of receiving Office and "PCT International Application" 



Applicant's or agent's file reference sON Y-747PC 
(if desired) (12 characters maximum) 



Box No. I TITLE OF INVENTION 

METHODS AND APPARATUS FOR COLOR DEVICE CHARACTERIZATION. IMAGE COLOR CORRECTION. 
M™nsS^ T£^^^^^^ ^MAGE DAm AND RBCX)RDING MEDIUM THEfaEFC 



Box No. n APPLICANT 



Name and address: ^am/7v name followed by given name; for a legal entity, full official designation. 

The address must include postal code and name of country. The country of the address indicated in this 

Box is the appUcant's State (that is, country) of residence if no State of residence is indicated below.) 

SONY CORPORATION 

7-35. Kitashinagawa 6-chome 

Shinagawa-ku. Tokyo 141-0001 

JAPAN 



State (that is, country) of nationality: 
JP 


State (that is, country) of 
JP 


residence: 


j^L^iL^yp^^ n ^.^^^ 5<1 □ ^^^^^^ □ Sli^^»goil 


Box Na m FURTHER APPUC ANT(S) AND/OR (FURTHER) INVENTOR(S) 





j I This person is also inventor. 



Telephone No. 
(03) 5448-2617 



Facsimile No. 
(03) 5448-3063 



Teleprinter No. 



IN ante ana autli ess. (ramify name juuvww uy gfi-tsr* yu-r - ^-^o" jj o 

The address must include postal code and name of country. The country of the address indicated in this 
Box is the appUcant's State (that is, country) of residence if no State of residence is indicated below.) 

NISHIO, Kenichi 
do Sony Corporation 
7-35, Kitashinagawa 6-chome 
ShInagawa-ku, Tokyo 141-0001 
JAPAN 



This person is: 

I I applicant only 

applicant and inventor 



I I inventor only (If this check-box 
is marked, do not fill in below.) 



State (that is, country) of nationality: 
JP 



State (that is, country) of residence: 
JP 



Further applicants and/or (fiirther) inventors are indicated on a continuation sheet. 



Box Na IV AGENT OR COMMON REPRESENTATIVE; OR ADDRESS FOR CORRESPONDENCE 



The person identified below is hereby/has been appointed to act on behalf 
of the applicant(s) before the con^)etent Internation al Authorities as: 

Name and address: (Family name followed by given name; for a legal entity, full official 
designation. The address must include postal code and name of country.) 

SAMMUT, Charles P. 
LIMBACH & LIMBACH LLP. 
2001 Ferry Building 
San Francisco, C A 94111 
United States of America 



IXI agent | | common representative 



Telephone No. 
(415) 433-4150 



Facsimile No. 
(415) 433-8716 



Teleprinter No. 



□ 



Address for correspondence: Mark this check-box where no agent or common representative isAias been appointed and the 



space 



above is used instead to indicate a special address to vvhich correspondence should be sent. 



Form PCT/RO/101 (first sheet) (July 1998; reprint January 2000) 



LegalStar 2000. Fonn PCTREQ 



See Notes to the request form 



Sheet No- 



Continuation of Box No. Ill FURTHER APPLICANTS AND/OR (FURTHER) INVENTOR(S) 


if none of the following sub-boxes is used, this sheet is not to be included in the request. 


Name and address: (Family name followed by given name; for a legal entity, full official designation. 
The address must include postal code and name of country. The country of the address indicated in thi 
Box is the applicant's State (that is, country) of residence if no State of residence is indicated below.) 

MIZUTANI. Eiji , 

c/o Sony Corporation 

7-35, Kitashinagawa 6-chonne 

Shinagawa-ku» Tokyo 141-0001 

Japan 


^ This person is: 

1 1 applicant only 

[Xl applicant and inventor 

1 1 inventor only (If this check-box 
is marked, do not fill in below.) 


State (that is, country) of nationality: 
US/JP 


State (that is, country) of residence: 
US 


This person is applicant | | all designated 1 1 all designated States except R7] the United States 1 | the States indicated in 
for the purposes of: 1 1 States 1 1 the United States of Amenca 1^ ofAmencaonly 1 — 1 the Supplemental Box 


Name and address :^f<z/nf(K name followed by given name; for a legal entity, full official designation. 
The address must include postal code and name of country. The country of the address indicated in this 
Box is the applicant's State (that is, country) of residence if no State of residence is indicated below.) 


This person is: 

{ 1 applicant only 

1 1 applicant and inventor 

1 I inventor only (If this check-box 
is marked, do not fill in below.) 


State (that is, country) of nationality: 


State (that is, country) of residence: 


This person is applicant 1 1 all designated 1 1 all designated States except | | the United States 1 1 the States indicated in 
for the purposes of: 1 1 States 1 1 the United States of America 1 — 1 of America only 1 — 1 the Supplemental Box 


Name and address: (Family name followed by given name: for a legal entity, full official designation. 
The address must include postal code and name of country. The country of the address indicated in this 
Box is the applicant's State (that is, country) of residence if no State of residence is indicated below.) 


This person is: 

1 1 applicant only 

1 1 applicant and inventor 

1 1 inventor only (If this check-box 
is marked, do not fill in below.) 


State (that is, country) of nationality: 


State (that is, country) of residence: 


This person is applicant 1 1 all designated 1 1 all designated States except | 1 the United States 1 1 the States indicated in 
for the purposes of: 1 1 States 1 1 the United States of America 1 1 of America only 1 — 1 the Supplemental Box 


Name and address: (Family name followed by given name: for a legal entity, full official designation. 
The address must include postal code and name of country. The country of the address indicated in this 
Box is the applicant's State (that is, country) of residence if no State of residence is indicated below.) 


This person is: 

1 1 applicant only 

1 1 applicant and inventor 

1 1 inventor only (If this check-box 
is marked, do not fill in below.) 


State (that is, country) of nationality: 


State (that is, country) of residence: 


This person is applicant 1 1 all designated 1 1 all designated States except 1 1 the United States | 1 the States indicated in 
for the purposes of: 1 1 States 1 1 the United States of America 1 1 of America only 1 — 1 the Supplemental Box 


1 1 Further applicants and/or (further) inventors are indicated on another continuation sheet. 



Form PCT/RO/101 (continuation sheet) (July 1998; reprint January 2000) Legaistar 2000. Form pctreq ^^^^^ request form 



Box No.V 



DESIGNATION OlTSTATES 



OFSTi^ 



Sheet No. . . .3. 



The following designations are hereby made under Rule 4,9(a) (mark the applicable check-boxes: at least one must be marked): 
Regional Patent 

□ AP ARIPO Patent: GH Ghana, GM Gambia, KE Kenya, LS Lesotho, MW Malawi, SD Sudan, SL Sierra Leone, SZ 

Swaziland, TZ United Republic of Tanzania, UG Uganda, ZW Zimbabwe, and any other State which is a Contracting State of 
the Harare Protocol and of the PCT 

□ EA Eurasian Patent: AM Armenia, AZ Azerbaijan, BY Belarus, KG Kyrgyzstan, KZ Kazakhstan, MD Republic of Moldova, 
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METHODS AND APPARATUS FOR COLOR DEVICE CHARACTERIZATION 



5 Background of the Invention 

1 . Field of the Invention 

The present invention relates to color correction of images, 
and in particular relates to methods and apparatus for color device 
characterization, correcting color of a color image obtained by an 
10 electronic camera, transmission of color corrected data and a 

recording medium having recorded thereon a color corrected data of 
a color image. 

2. Description of the Related Art 

There are known in the art two approaches for performing 

15 color correction for an image captured by an electronic color camera. 

A first approach is to directly convert the camera's output data into 
the same color space as camera's output color space with improved 
color accuracy. For instance, when the camera's output space is 
YUV (defined below), the color correction model's input and output 

20 spaces are YUV, but with higher color accuracy at the output YUV. 

Hereinafter, this approach will be referred to as "device specific 
color correction." "YUV" is a color representation which is typically 
used in connection with video transmission. "Y" represents 
luminance, "U" represents a component of colors corresponding 

25 approximately to blue minus yellow, while "V" represents 

component of colors corresponding approximately to red minus 
cyan. Other color representations include "RGB" (red, green blue) 
and "YCC" where YCC indicates any color space consisting of one 
luminance factor and two chrominance factors. The above- 

30 described YUV is one example of a YCC color representation. YIQ, 
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is another example of a YCC color representation, where "I" is a 
component of colors corresponding approximately to yellow-ish red 
minus light blue, while "Q" is a component of colors corresponding 
approximately to violet minus green-ish yellow. YCbCr (luminance, 
chrominance-blue, chrominance-red is yet another example of a YCC 
color representation. 

A second approach for color correction is divided into two 
steps. First, the camera's output color space is once converted into 
a colorimetric space (XYZ or L*a*b*), the space which is specified 
by CIE (International Illumination Committee, originally published in 
the French language). A colorimetric color space attempts to 
describe color of natural objects perceived by human eyes. Color 
data described in a colorimetric space corresponds to the original 
color of the images taken by an electronic color camera. In a second 
step, the color data converted into such a colorimetric space is then 
converted into an output device's (computer monitor or printer) 
specific color space, where the entire image path can provide 
colorimetrically correct color reproduction. This two-step concept is 
incorporated in the ICC (International Color Committee) color profile 
format, which is commonly used in computer image processing 
where relatively high accuracy is required in color reproduction. In 
the context of the color correction based on ICC's concept, the 
intermediate colorimetric space is often referred to as a "device 
independent color space." Therefore, this color correction approach 
is referred to as "device independent color correction." 



To perform the above-described device-independent color 
correction, one should attempt to determine an inverse function of a 
camera's color transformation. This step is called "color device 
characterization" for an electronic camera, and such characterization 
is employed to correct color of images obtained by the electronic 
camera. The conventional practice for determining the inverse 
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function of an electronic camera's color transformation is to apply a 
method widely used for determining an inverse function of image 
scanners' color transformation. This approach has been considered 
reasonable, because most color cameras have signal processing 
5 similar to that of image scanners. 

Figure 1 shows the signal-processing diagram that is used in 
both image scanners and most electronic color cameras. In Figure 1 , 
input data of image sensor 104 of camera 100 is assumed to be in 
colorimetric space XYZ, which is provided to inputs 101, 102 and 
10 103, respectively. The output space of image sensor 104 is RGB, 

while the camera's output space is YCC. Camera output is provided 
at outputs 110, 111, and 112, corresponding to Y, C and C, 
respectively. 

A simple reverse-model, shown in Figure 2, can be used to 

1 5 approximate the inverse function of the color transformation formed 

by the image sensor camera 100 shown in Figure 1 . In Figure 2, 
each block should perform inverse conversion of corresponding 
blocks in Figure 1. The pairs of corresponding blocks are 104-124, 
151-161, 152-162, 153-163 and 109-122. Specifically, matrices 

20 104 and 109 should be inverse matrices of 1 24 and 122 

respectively. Tone reproduction curves ("TRCs"), which are usually 
implemented by one-dimensional lookup tables {"1D-LUTs") 151, 
152, and 153, should be inverse functions of ID-LUTs 161, 162 
and 1 63, respectively. Since this color correction model consists of 

25 matrices and TRCs, it is called a "TRC-matrix model." 

As s.iown in Figures 1 and 2, the color transformation of 
image sensor 104 is linearly approximated by a 3x3 matrix 124. 
According to color science for electronic cameras, an image sensor's 
spectral analysis characteristics ideally should be a linear 

30 combinations of color matching functions xyz specified by CIE, that 

is, a set of spectral functions which represent human eye's color 
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sensitivity. Since all color cameras are designed to target these ideal 
characteristics, in many cases a 3x3 matrix can well approximate an 
image sensor used in electronic cameras, thereby providing a 
reasonably accurate inverse function of the camera's color 
5 transformation. This TRC-matrix model approach, however, cannot 

perform an accurate color correction under some circumstances. 

Approximation accuracy 

When an image sensor's color transformation cannot be well 
approximated by a 3x3 matrix in the TRC-matrix model (124 in 
Figure 2), color correction cannot be accurate. This situation can 
likely occur in many entry-level electronic cameras. This is also the 
case when the camera's signal processing does not match the 
configuration shown in Figure 1, such that the configuration of 
Figure 2 can no longer be a reverse model of the camera. For 
instance. Figure 3 is a block diagram which illustrates signal 
processing typically used in a CMYG (Cyan, Magenta, Yellow, and 
Green, also referred to as "complementary color") electronic color 
camera. At first glance. Figure 2 cannot work as a reverse model for 
the signal processing of Figure 3. In addition, it is not easy to 
construct a simple reverse model for the signal processing performed 
by the system of Figure 3, due to structural complexity of a CMYG 
camera's signal processing. 

Illuminant dependency 

In the case of an electronic camera, whether it be a still 
25 camera or one intended to record a succession of still or moving 

images, there is a somewhat unique problem, more specifically, that 
of illuminant dependency of color transformation. While an image 
scanner captures images using one fixed built-in illuminant all the 
time, an electronic camera captures images under a variety of 
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illuminants, for example, outdoors at dawn, mid-day or late 
afternoon, indoors with fluorescent, incandescent or other types of 
artificial lighting, a combination of artificial lighting, or even a 
combination of natural (outdoor) and artificial lighting. In an 
electronic camera, there is a mechanism to estimate the illuminant at 
the time of image capturing, wherein the camera's gain settings at 
R, G, and B channels are adjusted according to the on-camera 
illuminant estimation, so that the final output image has R = G = B 
with respect to a white object. This operation is called "white 
balancing," and the gain adjustment is placed between image sensor 
104 and 1 D-LUTs 105 in Figure 1, although it is not shown in the 
diagram. Usually, color transformation of a camera changes 
depending on the setting of white balancing. In other words, an 
electronic camera has illuminant dependency in its color 
transformation. It means color distortion or error caused by an 
electronic camera changes depending on illuminant. Therefore, for 
highly accurate color correction, it is necessary to prepare for 
inverse functions of the camera's color transformation under all 
possible illuminants, which is impractical. One existing solution for 
this problem is to prepare for a set of inverse functions (likely in a 
form of ICC color profile, using either a TRC-matrix model or a three- 
dimensional lookup table) of color transformation under several 
representative illuminants. A user then has to manually select from 
the set of preset inverse functions, considering which inverse 
function is the best fit to the illumination under which the user 
actually captured the image. This not only is a burden for the user, 
but also can result in inaccurate color correction if the user's 
selection is inappropriate. Furthermore, inaccurate correction can 
also occur when the actual illumination used for the image capture 
falls between the preset inverse functions. This is a limitation of 
the lookup-table based (TRC-matrix) method. Another limitation of 
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the look-up table based method is its inability to accurately correct 
color under multi-illuminant conditions. An alternative to the TRC- 
matrix nnethod is the three dimensional lookup table {"3D-LUT"). 
Although such a 3D-LUT is more accurate than the TRC-matrix, the 
size of the correction model is huge, and it is, as with the TRC- 
matrix method, incapable of accurately correcting color under multi- 
illuminant conditions. Thus, it would be desirable to provide a 
method and apparatus for accurately correcting the color of images 
taken by an electronic camera under arbitrary illuminants. 

Summary of the Invention 

It is an object of the invention to provide a method and 
apparatus for correcting the color of images obtained by an 
electronic camera under a variety of illuminants. 

It is also an object of the invention to provide a method and 
apparatus for characterizing a color device. 

It is an additional object of the invention to provide a method 
and apparatus for correcting the color of images obtained by an 
electronic camera under a combination of illuminants. 

It is yet another object of the invention to provide a recording 
medium having recorded thereon data representative of color 
corrected color images. 

It is yet an additional object of the invention to provide a 
method of transmitting data representative of color corrected color 
images- 

It is a feature of the invention to correct color of a color image 
by utilizing a neural network as a function approximator to overcome 
the limitations of the look-up table method in determining a 
correction to data representative of the color image. 

It is a feature of the invention to correct color of a color image 
by utilizing a multilayer perceptron ("MLP"), a form of neural 
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network, to determine a correction to data representative of the 
color image. 

It is yet another feature of the invention to correct color of a 
color image by utilizing a coactive neuro-fuzzy inference system 
("CANFIS") model to determine a correction to data representative of 
the color image. 

It is yet a further feature of the invention to perform an 
automatic color correction of a color image obtained by an electronic 
camera in the case of multi-illuminant conditions. 

It is yet another feature of the invention to convert image data 
into XYZ space, thereby providing better accuracy in white point and 
in color under sophisticated off-camera illumination estimation. 

It is yet an additional feature of the invention to easily realize 
color constancy by modifying or replacing a training data set. 

It is an advantage of the invention to a higher color correction 
accuracy of images obtained by an electronic camera than is 
generally possible with a conventional TRC-matrix. 

it is a further advantage of the invention to allow electronic 
camera users to concentrate on composition instead of spending 
time on color correction during a photo shoot, but also at the time of 
displaying and/or printing a photo. 

According to one aspect of the invention, there is provided a 
method of correcting color of a color image obtained by an electronic 
camera, comprising the steps of determining, using a neural 
network, a correction to data representative of the color image 
based upon an estimated iliuminant of the color image; and applying 
the correction to the data representative of the color image, wherein 
the iliuminant comprises multiple sources of illumination. 

According to another aspect of the invention, there is 
provided a method of correcting color of a color image obtained by 
an electronic camera, comprising the steps of determining, using a 
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multilayer perceptron model, a correction to data representative of 
the. color image based upon an estimated illuminant of the color 
image; and applying the correction to the data representative of the 
color image, wherein the illuminant comprises multiple sources of 
5 illumination. 

According to a further aspect of the invention, there is 
provided a method of correcting color of a color image obtained by 
an electronic camera, comprising the steps of determining, using a 
coactive neuro-fuzzy Inference system model, a correction to data 

10 representative of the color image based upon an estimated illuminant 

of the color image; and applying the correction to the data 
representative of the color image, wherein the illuminant comprises 
multiple sources of illumination. 

According to yet another aspect of the invention, there is 

15 provided an apparatus for correcting color of a color image obtained 

by an electronic camera, comprising a neural network for 
determining a correction to data representative of the color image 
based upon an estimated illuminant of the color image and for 
applying the correction to the data representative of the color image, 

20 wherein the illuminant comprises multiple sources of illumination. 

According to yet a further aspect of the invention, there is 
provided an apparatus for correcting color of a color image obtained 
by an electronic camera, comprising a multilayer perceptron model 
for determining a correction to data representative of the color image 

25 based upon an estimated illuminant of the color image, and for 

applying the correction to the data representative of the color image, 
wherein the illuminant comprises multiple sources of illumination. 

According to another aspect of the invention, there is 
provided an apparatus for correcting color of a color image obtained 

30 by an electronic camera, comprising: a coactive neuro-fuzzy 

inference system model for determining a correction to data 
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representative of the color image based upon an estimated illuminant 
of the color image, and for applying the correction to the data 
representative of the color image, wherein the illuminant comprises 
multiple sources of illumination. 
5 According to a further aspect of the invention, there is 

provided a recording medium having recorded thereon color 
corrected data of a color image obtained by an electronic camera, 
the recording medium being prepared by the steps of determining, 
using a neural network, a correction to data representative of the 
10 color image based upon an estimated illuminant of the color image; 

applying the correction to the data representative of the color image, 
wherein the illuminant comprises multiple sources of illumination; 
and recording on the recording medium data representative of the 
corrected data. 

15 According to yet another aspect of the invention, there is 

provided a recording medium having recorded thereon color 
corrected data of a color image obtained by an electronic camera, 
the recording medium being prepared by the steps of determining, 
using a multilayer perceptron model, a correction to data 

20 representative of the color image based upon an estimated illuminant 

of the color image; applying the correction to the data representative 
of the color image, wherein the illuminant comprises multiple 
sources of illumination; and recording on the recording medium data 
representative of the corrected data. 

25 According to yet another aspect of the present invention, 

there is provided a recording medium having recorded thereon color 
corrected data of a color image obtained by an electronic camera, 
the recording medium being prepared by the steps of determining, 
using a coactive neuro-fuzzy inference system model, a correction to 

30 data representative of the color image based upon an estimated 

illuminant of the color image; applying the correction to the data 



wo 00/65847 



PCT/USOO/11504 



-10- 



10 
15 
20 
25 
30 

) 



representative of the color image, wherein the illuminant comprises 
multiple sources of illumination; and recording on the recording 
medium data representative of the corrected data. 

According to an aspect of the invention, there is provided 
a method of transmitting color corrected data of a color image 
obtained by an electronic camera, comprising the steps of 
determining, using a neural network, a correction to data 
representative of the color image based upon an estimated illuminant 
of the color image; applying the correction to the data representative 
of the color image, wherein the illuminant comprises multiple 
sources of illumination; and transmitting data representative of the 
corrected data. 

According to another aspect of the invention, there is 
provided a method of transmitting color corrected data of a color 
image obtained by an electronic camera, comprising the steps of: 
determining, using a multilayer perceptron model, a correction to 
data representative of the color image based upon an estimated 
illuminant of the color image; applying the correction to the data 
representative of the color image, wherein the illuminant comprises 
multiple sources of illumination; and transmitting data representative 
of the corrected data. 

According to a further aspect of the invention, there is 
provided a method of transmitting color corrected data of a color 
image obtained by an electronic camera, comprising the steps of: 
determining, using a coactive neuro-fuzzy inference system model, a 
correction to data representative of the color image based upon an 
estimated illuminant of the color image; applying the correction to 
the data representative of the color image, wherein the illuminant 
comprises multiple sources of illumination; and transmitting data 
representative of the corrected data. 
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According to an aspect of the invention, there is provided a 
method of recording image data obtained by an electronic camera, 
comprising the steps of: capturing a color image and generating data 
representative of the image; estimating an illuminant for the captured 
color image and generating data representative of the estimated 
illuminant; and recording the data representative of the image with 
the data representative of the estimated illuminant. 

According to a further aspect of the invention, there is a 
provided a method of transmitting image data obtained by an 
electronic camera, comprising the steps of: capturing a color image 
and generating data representative of the image; estimating an 
illuminant for the captured color image and generating data 
representative of the estimated illuminant; and transmitting the data 
representative of the image with the data representative of the 
estimated illuminant. 

These and other objects, features and advantages will become 
apparent when considered with reference to the following 
description and the accompanying drawings. 

Brief Description of the Drawings 

Figure 1 is a block diagram of a signal processor used in both 
image scanners and electronic color cameras. 

Figure 2 is a reverse signal processor for use in connection 
with the signal processor of Figure 1 . 

Figure 3 is a block diagram of a signal processor typically used 
in a comple.nentary color electronic camera. 

Figure 4A is a block diagram of an embodiment of a color 
correcting apparatus of the present invention in which a neural 
network is utilized to generate an inverse function (or mapping) of an 
electronic camera's color transformation. 
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Figure 4B is a diagram of an embodiment of a color correcting 
apparatus of the present invention in which multilayer perceptrons 
are utilized to generate an inverse function (or mapping) of an 
electronic camera's color transformation. 

Figure 5 is a diagram of an embodiment of a color correcting 
apparatus of the present invention in which single multilayer 
perceptron are utilized to generate a color corrected output space 
converted image. 

Figure 6A is an illustration of a dogleg trajectory having a 
piecewise linear curve determined by the Cauchy step and the 
Newton step, connecting ©^^w and ©Newton by the route through ©cauchy 

Figure 6B is an illustration of a dogleg step which leads to 
^nexu where the dogleg step is a piecewise linear approximation to a 
restricted Levenberg-Marquardt step within the trust region. 

Figure 7 is a graph which illustrates a comparison in color 
difference under a D65-illuminant between a single multilayer 
perceptron approach and a conventional look-up-table based (TRC- 
matrix) approach. 

Figure 8 is a diagram of the architecture of a CANFIS neuro- 
fuzzy model of the present invention. 

Figure 9 is a graph which illustrates representative illuminants 
and related "natural" and "flourescent" curves. 

Figure 10 is a graph which illustrates fuzzy partitioning by four 
bell-shaped membership functions on a transformed white balance 
plane. 

Figure 1 1 illustrates a data distribution on original coordinates 
of a white balance plane. 

Figure 1 2 illustrates a data distribution on normalized 
coordinates of the white balance plane. 

Figure 13 illustrates a neural network transformed distribution 
on transformed coordinates of the white balance plane. 
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Detailed Description of the Preferred Embodiments 

To solve the problems described above, a neural network 
(e.g., a MLP) or a CANFIS neuro-fuzzy model is utilized. Although 
the invention is applicable to both "device specific" and "device 
independent" approaches, the embodiments disclosed herein are 
described herein in relation to device independent color correction. 
It is to be understood that a MLP is just one type of well known 
neural network. Furthermore, where a neural network is specified in 
herein, a MLP or other type of neural network may be utilized. 

A single-illuminant color correction using an MLP. 

As discussed above, when a color correction using a TRC- 
matrix model (Figure 1 ) cannot be accurate even under one fixed 
known illuminant, the cause is likely to be non-ideality, nonlinearity 
or structural complexity in the electronic camera. A neural network 
as shown in Figure 4A or a MLP model as shown in Figure 4B can be 
used instead of the TRC-matrix model to improve the correction 
accuracy. In Figure 4A any neural network may be utilized, for 
example, the MLP of Figure 4b. With the neural network (e.g., MLP) 
model, the correction can be based on one fixed illuminant, thus 
hereinafter referred to as "single-illuminant" color correction. A 
color correction model using a neural network (e.g., MLP) had been 
used only for imaging devices such as printers, whose color 
transformation is highly nonlinear and whose white point 
(corresponding to the color of illumination in case of an electronic 
camera) is considered fixed. 

A Y input 1, a C input 2 and a C input 3 provide image data to 
neural network 4 as shown in Figure 4A. Image data is provided 
from an electronic camera. Alternatively, image data may be 
provided to inputs 1 , 2 and 3 from a reproducing apparatus and/or a 
transmission medium. In the latter instances, since illumination 
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information necessary for color correction is only available from the 
electronic camera, such illumination information should be recorded 
and/or transmitted with the image data, or should otherwise be 
provided to the neural network 4. In the case of transmission or 
recording of image data from an electronic camera, many image data 
formats include unused fields or spaces for use by image printers. 
Such fields or spaces are typically available to record the date of a 
photo shoot, a camera model designation or a camera manufacturer 
designation. Therefore, one or more unused field or space may be 
allocated to illumination information, such that the image data is 
recorded and/or transmitted together with the illumination 
information, but before any color correction has been applied to the 
image data. 

The neural network 4 provides an X output 5, a Y output 6 
and a Z output 7 in the colorimetric space XYZ. As shown in Figure 
4B, the MLP has three input nodes, Y, U and V, designated by 
reference numerals 11,12 and 13, respectively, for camera's output 
values (RGB or YCC). The MLP also has three output nodes X, Y 
and Z, designated by reference numerals 14, 15 and 16, 
respectively. Output nodes X, Y and Z provide either colorimetric 
values (XYZ or L*a*b*) for device-independent color correction, or 
target output values in RGB or YCC for device-specific color 
correction. These colorimetric or color-corrected output values may 
be provided to image output devices, such as printers or display 
monitors, via the output device's characterization model. The values 
may also be directly provided to image output devices. 
Alternatively, these values, encoded and/or compressed versions 
thereof, may be recorded by a recording apparatus onto a recording 
medium such as a magnetic disc, an optical disc, a magneto-optical 
disc, or a sold state memory, or may be transmitted through a 
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transmission medium such as the Internet, telephone lines, dedicated 
lines, radio frequency or an optical medium. 

For optimizing an MLP's parameters, a set of input-output 
samples, called a "training" data set, is prepared that includes 
5 representative color samples, pairs of inputs YUV and desired 

outputs XYZ. Then a given task is formulated as a nonlinear least 
squares problem, in which, the objective is to optimize an MLP's 
parameters by minimizing a squared error measure between the 
desired outputs and the model's outputs. 
O The objective function E(.), the squared error measure, can be 

expressed by residual vectors r{6) as: 

EiO) = n/2)r^{e)r(e) 
The parameter optimization is carried out iteratively with the aim of 
eventually making the model reproduce the desired outputs. This 
iterative process is referred to as "training" (or "learning") in the 
neural network literature. 

Besides the "training" data set, an MLP's approximation 
capacity Is often measured by using a different data set, called a 
"checking" data set, which is not employed for parameter 
adjustment. One important advantage of neural networks, and in 
particular, those of an MLR, are the results obtained for input data 
outside of the training data set. 

In the posed "device-independent" color correction problem, 
the MLR is trained to form a mapping from a camera's response 
signals (e.g., RGB or YCC) to colorimetric values (e.g., XYZ or 
L*a*b*). In the "device-specific" color correction problem, the 
MLP's desired mapping is from error-corrupted camera's response 
signals (RGB or YCC) to their associated corrected RGB or YCC 
signals. In any event, those input-output samples are collected by 
using the standard Macbeth ColorChecker or Munsell color 
patches, which are evenly distributed in the entire color space. 
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Note that if the data are collected under a certain single 
illuminant, then the task beconnes the so-called single-illunninant 
problem, which is ubiquitous in color correction for images captured 
by image scanners. 



5 Multi-illuminant color correction using an MLP 

Referring now to Figure 5, an MLP model is used to achieve 
an automatic color correction for images captured by an electronic 
camera under arbitrary illuminant (at the time of image capture). The 
MLP model is trained so that it automatically adjusts its mapping, an 

10 inverse function of the camera in a case of device-independent 

correction, depending on the illuminant used at the time of image 
capture. This is hereinafter referred to as "multi-illuminant" color 
correction. Since it is difficult to exactly know what illuminant was 
used at the time of image capture, the model instead takes 

1 5 advantage of the electronic camera's on-camera illumination 

estimation data (or white balancing data), R/G and B/G values for 
instance, where R, G, and B are measured signal values for 
illumination color, as illustrated in the upper portion of Figure 5. In 
Figure 5 "Pana white" refers to a trademark for a "white" flourescent 

20 bulb sold in Japan by Matsushita Electric Industrial Co. Ltd. of 

Osaka, Japan. "Mellow d" refers to a trademark which identifies a 
"daylight" type of flourescent bulb sold in Japan by Toshiba Lighting 
& Technology Co. of Tokyo, Japan. It is to be understood that 
Figure 13 can be used in place of the upper portion of Figure 5 

25 (White balance plane) 

The MLP used for the multi-illuminant color correction should 
have additional nodes to input the illumination information. For 
instance, two additional input nodes can be used because 
illumination chromaticity can be described by a set of two values. 

30 Consequently, the MLP model for this task has five input nodes B/G, 
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R/G, Y, U and V, designated by reference numbers 17, 18, 19, 20 
and 21, respectively, and three output nodes X, Y and Z, designated 
by reference nunaerals 22, 23 and 24, respectively, as shown in 
Figure 5. The outputs X, Y and Z, or transformed, normalized, 
5 encoded and/or compressed versions thereof, may be recorded by a 

recording apparatus onto a recording medium such as magnetic disc, 
an optical disc, a magneto-optical disc, or a solid state memory or 
may be transmitted through a transmission medium such as the 
Internet, telephone lines, dedicated lines, radio frequency or an 
10 optical medium. 

In accordance with one embodiment of the invention, the 
following steps are taken to train a multi-illuminant color correction 
model: 

(1) Measure an electronic camera's output values and 

1 5 colorimetric values {for device-independent correction) 

of a set of color samples under several representative 
illuminants. The total size of training data set is 
therefore N*M, where N is the number of color samples 
and M is the number of representative illuminants under 

20 consideration. 

(2) For each illuminant, measure gain data of the RGB 
channels as applied to balance white, or instead, 
measure signal values of RGB channels for illuminant 
color. 

25 (3) Convert the measured three values onto a set of two 

chromaticity values, for instance, R/G and B/G. The 
data may be further processed to better characterize 
the illuminant information, as described below with 
respect to a method for constructing fuzzy membership 

30 functions for the CANFIS neuro-fuzzy model. 
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(4) An MLP is trained using the data set (1), as described 
above. At the same time, the data set (3), that is, the 
illuminant information under which data (1) is 
measured, are provided to the input nodes 17 (B/G) and 
18 (R/G). 

After the above described steps have been completed, the 
MLP model outputs the original scene's colorimetric values by having 
camera's output values and the illuminant information at its input. 
The camera's illuminant dependency of color transformation is 
automatically compensated by referring to the illumination 
information. When the illuminant actually used at the time or times 
of image capture does not coincide to any of the representative 
illuminants used for the MLP training, the MLP automatically 
approximates the inverse function of the camera's color 
transformation between illuminants. 

Method for training MLP models 

Due to the nonlinearity of MLPs, an efficient iterative method 
is desirable for solving the NN nonlinear least squares problem posed 
in above section. Numerical tests have revealed that a dogleg trust- 
region implementation of the Levenberg-Marquardt-type algorithm is 
suitable for the problem to be solved. In further detail, dogleg 
methods produce an efficient piecewise linear approximation of the 
restricted Levenberg-Marquardt step within the trust region. 

Figures 6A and 6B illustrate how the dogleg step 
approximates the trust region step over a two dimensional subspace 
spanned by the Cauchy step and the Newton (or Gauss-Newton) 
step. The crosspoint between the piecewise linear dogleg trajectory 
(highlighted in Figure 6A) and the trust-region boundary yields one of 
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the next three steps: 

1 , Restricted Cauchy step: 



^ dot n 9now 
^RC — -^now— 



when the trust radius R^^ is smaller than the length of the Cauchy 
step Scauchv which is given by 



r 



C ^ - 9no^now ^ 

*^Cauchy — ~ ynow 



7 now 



where g and H denote the gradient vector and the Hessian matrix, 
respectively. 

2. Newton step: 

^Nawton = ~^^noy^i 



now 



when the trust radius is larger than or equal to the length of the 
Newton step. 



10 3. Dogleg step: 



^Dogleg — ^Cauchy ^novX^Nawton ^Cauch^ 

when the trust radius is in between the Cauchy step and the Newton 
step. One of those steps is employed at each iteration for optimizing 
the model by the iterative scheme: 
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where d is a direction vector and rj is some positive step size 
regulating to what extent to proceed in that direction. The 
parameter h in Figure 6A can be determined in a straightforward 
calculation: 

_ ^now ~ ^Cauchy^Cauchy 

''now ^ J^HZ 

Newton ~ ^Cauchyi ^Cauchy )/ Pnow 



where 



Pnow (^/Vwton ^Cauch^^i^Nevirton ^Cauchyi^now 

(^Newton ^Cauch))^ ~ Newton ^Newtof)i^Cauchy^Cauchy 



The power of the dogleg trust-region strategy resides in ease 
for a smooth transition between the steepest descent method (global 
convergence property) and Newton's method (fast local convergence 
property), making the algorithm very robust. For instance, when a 
10 next step is not satisfactory, the dogleg algorithm shortens the step 

length and deflects it towards the steepest descent direction 
simultaneously and efficiently based on a simple trust-region control. 



m 



wo 00/65847 PCTAJSOO/1 1504 



-21- 

Table 1 below compares the convergence speed of three 
representative nonlinear optimization algorithms in a single-illuminant 
(D65-illuminant) problem: 







25 training data 


51 checking data 


Stopped epoch 


5 


lO lookup table with 
a 3 X 3 matrix transformation 


0.015900 


0.025576 


N/A 




MLP with 7 hidden units by steepest descent- 
based pattern by-pattern learning 




U.UibobZ 


1 nnn nnn 

(10,818.61 sec.l 




MLP with 10 hidden units by steepest descent- 
based pattern by-pattern learning 


0.004608 


0.015991 


1,000,000 


10 


MLP with 7 hidden units by Polak-Ribiere 
conjugate gradient batch learning 


0.005797 


0.018310 


2.400 
(114.65 sec.) 




MLP with 10 hidden units by Polak-Ribiere 
conjugate gradient batch learning 


0.004519 


0.016344 


2,000 


15 


MLP with 7 hidden units by a dogleg-driven 
Levenberg-Marquardt learning 


0.00552 


0.017947 


44 

(8.54 sec.) 



TABLE 1 



The performance comparison is in root mean squared error of scaled 
XYZ values between the TRC-matrix oriented model and the MLP 
models with three representative learning algorithms. Note in the 

20 last column "Stopped epoch" that the required execution time (in 

seconds) is shown parenthesized only the for (3x7x3) MLP 
models with seven hidden units. From Table 1 it is clear that for a 
single-illuminant color correction problem, the use of the dogleg 
driven Levenberg-Marquardt converges faster than the other tested 

25 learning techniques indicated in Table 1. Such performance 

comparison is further discussed in Color device characterization of 
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electronic cameras by solving adaptive networks nonlinear least 
squares problems, Eiji Mizutani (a co-inventor of the present 
application), Kenichi Nishio (the other co-inventor of the present 
application), Naoya Katoh and Michael Blasgen, 8th IEEE 
International Conference on Fuzzy Systenns (FUZZ-IEEE'99), Seoul, 
Korea, August 22-25, 1999, which paper and the figures therein are 
hereby incorporated by reference. Furthernnore, the dogleg trust- 
region algorithms are further discussed in Computing Powell's 
Dogleg Steps for Solving Adaptive Networks Nonlinear Least- 
Squares Problems, Eiji Mizutani (a co-inventor of the present 
application), 8th International Fuzzy Systenns Association World 
Congress (IFSA'99), Hsinchu, Taiwan, August 17-20, 1999, and in 
Powell's dogleg trust-region steps with the quasi-Newton augmented 
Hessian for neural nonlinear least-squares learning, Eiji Mizutani (a 
co-inventor of the present application). The IEEE International 
Conference on Neural Networks, Washington D.C., July 10-16, 
1 999, both of which papers and the figures therein are hereby 
incorporated by reference. 

Referring now to Figure 7, such Figure illustrates a comparison 
in color difference under a D65 illuminant between a single MLP 
approach and a conventional look-up table based (TRC-matrix) 
approach. Note that all 25 color differences (for 24 Macbeth color 
checkers and a "perfect-black"color sample) must be smaller than 
10. Figure 7 clearly illustrates an MLP based method effectively 
reduces the color difference down below an upper-limit threshold 
value (set equal to 10) for all 25 color samples under the D65 
illuminant, whereas a conventional look-up-table based (TRC matrix) 
method failed to do so. 
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Multi-illuminant color correction using a CANFIS neuro-fuzzy model 
A CANFIS neuro-fuzzy model is specially designed to perform 
automatic "multi-illuminant" color correction based on problem- 
specific knowledge, which is represented by a collection of fuzzy if- 
then rules: In particular, "IF-part" is expressed by fuzzy membership 
functions (MFs) and "THEN-part" is constructed by local-expert 
MLPs. This is the so-called "CANFIS with neural rules" model. 

In particular, four fuzzy rules are specified in accordance with 
fuzzy-partitioning on the two-dimensional transformed white balance 
plane, as explained below in relation to a method for constructing 
fuzzy membership functions. The resulting CANFIS neuro-fuzzy 
model consists of four fuzzy MFs (two for each axis) and four 
associated local-expert MLPs, as illustrated in Figure 8. The input 
vector is split into two vectors for IF-part (MFs) and THEN-part 
(MLPs): 

1 . Two inputs of on-camera illumination information fed 
into fuzzy MFs; 

2. Three inputs of camera's output signals (e.g., YUV) fed 
into local-expert MLPs. 

The CANFIS model computes a weighted sum of the outputs of 
local-expert MLPs as the final output vector A (e.g., X, Y, Z color 
signals) by 



where g; and Oj are the i-th firing strength and the i-th MLP's output 
vector, respectively. Each firing strength, given by the product of 
IF-part MF outputs, represents the extent to which the associated 
THEN-part MLP affects the final output. In this context, a set of IF- 
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part MFs plays a role as an Integrating unit that combines the 
outputs of fired THEN-part MLPs. 

Usually, the integrating unit is composed of bell-shaped 
differentiable fuzzy MFs and their neighboring MFs are set up to 
have sufficient overlap. Hence, all of the associated local-expert 
MLPs function complementarily to generate the final output in the 
equation for the value of A. On the other hand, if the integrating 
unit is constructed by non-overlapped rectangular MFs resulting in a 
switching unit, then only a single local-expert MLP contributes to the 
final output. Only one of the gj values becomes a "unit" with all of 
the others being "zero." Such a single non-zero g, is associated with 
the i-th "fired" local expert MLP. In other words, local expert MLPs 
function competitively rather than complementarily. Stated 
differently, the CANFIS concept basically resides in the synergism 
among fuzzy MFs and local-expert MLPs, ruling out such an extreme 
switching case. 

The CANFIS model has an advantage in that without 
increasing the number of fuzzy MFs (so that fuzzy rules can be held 
to meaningful limits), the model can increase learning capacity just 
by applying an appropriate architectural or algorithmic modification 
solely to the local-expert MLPs. For instance, such modification can 
simply adjust the number of MLP hidden nodes/layers. It should be 
noted that the CANFIS model can be viewed as a local-tuning RBFN 
(Radial Basis Function Network) model when the hidden layers of 
THEN-part MLPs are eliminated. Hence, the CANFIS model can be 
regarded as a generalized local-tuning neural network. 

Its disadvantages might be slower learning due to architectural 
complexity and a requirement of larger training data, compared with 
a single MLP model. Therefore, if the task has a small limited 
number of training data, then an MLP model might be a better 
choice. Experimental results obtained with the CANFIS model are 
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presented below in relation to a method for training a CANFIS neuro- 
fuzzy model for multi-illuminant color correction. 

The CANFIS with linear rules model is equivalent to the well- 
known Takagi-Sugeno (also called "TSK" or "Sugeno-type") fuzzy 
inference model. In the field of control engineering, where a quick 
response is important, this TSK model is by far the most commonly 
used fuzzy system. The quickness comes from a computational 
advantage since implementation of linear rules is much simpler than 
implementation of nonlinear rules. 

CANFIS modeling and CANFIS with linear rules are explained 
in detail at pages 369-400 and 572-592, of the text Neuro-Fuzzy 
and Soft Computing, A Computational Approach to Learning and 
Machine Intelligence, by Jyh-Shing Roger Jang, Chuen-Tsai Sun and 
Eiji Mizutani (co-inventor of the present application). Prentice Hall, 
Upper Saddle River, New Jersey 07458, 1997, third printing, which 
pages and the figures therein are hereby incorporated by reference. 

Method for constructing fuzzy membership functions for multi- 
illuminant color correction using the CANFIS neuro-fuzzy model 
described above. 
Fuzzy Partitioning 

In the CANFIS neuro-fuzzy model, fuzzy membership functions 
(MFs) partition the MF-input space into several local-expert's 
territories. Hence, data that come close to the center of a certain 
local-expert's area may not affect very much to the parameter- 
updatings of the other local experts. This local-tuning mechanism 
can make "local experts" learn locally optimal mainly for the vicinity 
of their own local regions. 

For the multiple-illuminant problem as illustrated in Figure 9, 
data in the halogen illuminant region should not make a significant 
impact on a local-expert MLP in the D90-illuminant region, which is 
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at the other end of the "natural-illuminant" curve on the white 
balance plane. Those two extreme regions must be somehow 
distinguished by an appropriate partitioning. Fuzzy MFs plays an 
important role in partitioning the illuminant territories on the white 
balance plane in a "fuzzy" (but not "crisp") fashion, as illustrated in 
Figure 10. In Figure 10, two representative illuminants, halogen and 
D65, are chosen to determine the initial center positions of the two 
MFs on the X-axis, which corresponds to the "natural-illuminant" 
curve. This implies that the location along that curve is selected as 
an important feature in the task. The two linguistic labels "A1 " and 
"A2" on the X-axis in Figure 10 signify the respective "Halogen" and 
"D65" where: 

The value/degree of A1-MF shows how close the illuminant is 
to the "Halogen" illuminant; and 

The value/degree of A2-MF shows how close the illuminant is 
to the "D65" illuminant. 

Clearly, only one feature may not be sufficient for describing 
each illuminant's region appropriately. Another important feature is 
the distinction between the "natural" and the "fluorescent" 
illuminants. Therefore, two corresponding MFs (B1 for "fluorescent" 
and B2 for "natural") are constructed on the Y-axis to represent 
closeness to the fluorescent and natural illuminants, where: 

The B1-MF value shows how close to the "fluorescent" 

illuminant-curve; and 

The B2-MF value shows how close to the "natural" illuminant- 
curve. ^ 
In this way, each of mixed illuminants along the "mixtures" curve in 
Figure 9 can be characterized by the degree of membership to the 
"natural" or "fluorescent" illuminant, as would be the case where an 
illuminant consisted of a 40 % natural illuminant and a 60 % 
fluorescent illuminant. In Figure 9, "Lupica ace" refers to a 
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trademark which identifies a "neutral" type of flourescent bulb sold 
in Japan by Mitsubishi Electric Co. of Tokyo, Japan. Since the X 
and Y axes in Figures 9 and 10 are not the same, a certain 
"nonlinear" coordinate transformation is needed. Such coordinate 
transformation is described below. 

Nonlinear Coordinate Transformation on the White-Balance Plane 

Figure 1 1 shows an actual data distribution of experimental 
data sets on the original R/G-B/G white balance plane. To conform 
Figure 1 1 to Figure 10, it is necessary to introduce an appropriate 
coordinate transformation. First, a simple linear-scaling is applied, 
resulting in Figure 12. Then a neural network nonlinear coordinate 
transformation is applied, yielding Figure 13, where Figure 13 has 
Cartesian coordinates that match those of Figure 10, although many 
other nonlinear transformations are possible. 

The importance of nonlinear coordinate transformation cannot 
be overemphasized because it allows simple "grid" partitionings. 
There are several advantages over "scatter" partitionings as are 
typically formed by a radial basis function network ("RBFN"). RBFNs 
are explained in detail at pages 238-246 and 369-400, and in 
particular, at Figure 13.3 on page 373 of said text Neuro-Fuzzy and 
Soft Computing^ A Computational Approach to Learning and 
Machine Intelligence, which pages and the figures therein are 
hereby incorporated by reference. 

In further detail, nonlinear coordinate transformation provides 
the following advantages: 

1 . Linguistic interpretability can be kept sufficiently high to 
match our intuitive understandings; and 

2. The required number of (basis) functions can be 
reduced. 
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Advantage #1 insinuates that clear linguistic labels may not be 
regdily put on each MF in the case of scatter or tree partitionings. 
Advantage #2 indicates, for instance, that in order to form nine local 
regions on the X-Y plane, the RBFN needs nine basis functions, 
whereas the CANFIS model requires six MFs (three MFs for each 
axis). Accordingly, the total number of function parameters differs. 
Further detail regarding partitioning is taught at pages 86 and 87 of 
said NeurO'Fuzzy and Soft Computing^ A Computational Approach to 
Learning and Machine intelligence, which pages and the figures 
therein are hereby incorporated by reference. 

Method for training a CANFIS neuro-fuzzy model for multi-illuminant 
color correction. 

Many nonlinear optimization algorithms can be employed for 
training the CANFIS neuro-fuzzy model described above in relation to 
a multi-illuminant color correcting using a CANFIS neuro-fuzzy 
model. The model consists of two major parameterized 
constituents: 

(1) "fuzzy MFs (IF-part);"" and 

(2) "local-expert MLPs (THEN-part)," 

as illustrated in Figure 8. There are many possibilities in applying 
optimization algorithms to update the parameters of these two 
constituents, for example: 

(Method A) Apply the same optimization algorithm to both IF- 

part MFs and THEN-part MLPs; or 

(Method B) Apply different optimization algorithms to IF-part 

MFs and THEN-part MLPs. 
These methods can be implemented in either a system-wise or a 
component-wise manner. For system-wise training, all components, 
MFs and MLPs, of the CANFIS neuro-fuzzy model are trained 
simultaneously as a system, whose final output is computed by the 
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above equation for the value of A as set forth above. For 
component-wise training, each MF or MLR is tuned/trained 
independently, and then all the components are put together to form 
the CANFIS neuro-fuzzy model, whose final output is computed by 
the above equation for the value of A as set forth above. 

To be more specific by referring to Figure 8, In the 
component-wise manner, each local expert MLP is usually trained to 
produce the desired final output vector A (i.e., X, Y, Z) as the output 
vector Oj. On the other hand, in the system-wise training manner, 
the final output vector A (as set forth above) is attempted to be 
made to match the desired triplet X, Y and Z, but each local expert 
MLP's output vector Oi may not be necessarily close to it; that is, 
each output vector Oj may not be interpretable. In Figure 8, the 
three output nodes of each local expert MLP are tagged by X, Y and 
Z, merely for the purpose of showing which node is related to which 
signal. 

Furthermore, there are three different parameter-updating 
modes, depending on treatment of training data: 

(1) online pattern-by-pattern updating mode; 

(2) block-by-block updating mode; and 

(3) batch updating mode. 

Therefore, there are many possible applications of any optimization 
algorithms to the CANFIS model. It is worth noting, however, that 
regardless of whether Method A or Method B is utilized, the 
component-wise training is questionable in obtaining highly accurate 
results because local-expert MLPs' outputs are optimized 
independently of the firing strengths of fuzzy MFs in spite of the 
dependence expressed in the above equation for the value of A as 
set forth above. Thus, the other system-wise training is of much 
greater practical importance, especially when the CANFIS model 
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needs to produce very accurate input/output mappings, as required 
in color correction of color images. 

In general, an important observation can be made in neuro- 
fuzzy systems that the parameters of MLPs are far from optimal 
values, compared with those of MFs, because the initial parameters 
of fuzzy MFs are determined based on problem-specific knowledge, 
whereas the initial parameters of local-expert MLPs are randomly 
initialized. Therefore, application of different optimization algorithms 
to MFs and MLPs (i.e., implementation of Method B) is possible: in 
particular, application of a faster algorithm to MLPs and a slower 
algorithm to MFs. A typical demonstration in this spirit is the hybrid- 
learning of ANFIS (i.e., single-output CANFIS with linear rules 
model), a combination of fast "linear" least squares and a slow 
steepest descent-type method, although it is restricted to "linear" 
consequent fuzzy systems. Such hybrid-learning is detailed at pages 
21 9-220 and 222-223 of said text Neuro-Fuzzy and Soft Computing, 
A Computational Approacli to Learning and l\Aachine Intelligence, 
supra, which pages and the figures therein are hereby incorporated 
by reference. 

Method A in the system-wise training manner can be 
considered in the same spirit for the CANFIS with neural rules 
model; for instance, by applying the steepest descent-type method 
to MFs in the "batch" mode and it to MLPs in the "online pattern-by- 
pattern" mode. This strategy can be improved in conjunction with a 
simple heuristic rule; for instance: 



Table 2 shows four representative results obtained with MLP 
and CANFIS models, when the checking error was minimized or at 
the preset limit iteration of 2,000,000. The column "# of Para." 
refers to the number of parameters and "RMSE" refers to the root 



Rule: Fix the parameters of fuzzy MFs at an early stage 
of training phase. 
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mean squared error. The columns labelled "# of bad samples" are 
most important, denoting the number of color samples whose color 
differences were greater than the upper-limit threshold value (set 
equal to 10). All of the four models were trained by using the 
widely-employed steepest descent-type method in the online pattern- 
by-pattern updating mode. 
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The results in Experiment #4 were obtained by using the 
following heuristic rule: 

Rule: Start updating the parameters of fuzzy MFs after 
500,000 epochs. 

Selection of output color space 

In the present disclosure, XYZ space is mainly used as an 
output color space of color correction models. As described earlier 
herein, the output color space can also be L*a*b* or XnYnZn 
(normalized XYZ space) for device-independent color correction. For 
device-specific color correction, it can be any color space that the 
color imaging device subject to the color correction outputs, or can 
even be a color space arbitrarily defined according to a user's 
specific requirement. The change of output color space can be 
achieved by simply replacing the target output data for color 
correction model's training data set. 

As discussed above in relation to llluminant dependency, an 
electronic camera's output values are white-balanced to keep 
R = G = B at illuminant color. XnYnZn and L*a*b*^ color spaces also 
have a similar operation in their definition, wherein tri-stimulus 
values are normalized by values for illuminant color. Such a color 
space can be denoted as a normalized color space. This operation 
represents the human eye's chromatic adaptation. 

As shown in Figure 1 , an electronic camera's output values 
are based on gamma-corrected RGB. Similar operations are found in 
the definition of L*a*b* color space, and such a color spaces can be 
considered to be a nonlinear color space. The nonlinear mapping is 
for a correlation with lightness (a measure for human's luminance 
perception), to better describe perceived color by human eyes rather 
than the physical property of color. 
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A non-normalized linear color space, such as XYZ, can provide 
an advantage when used as a multi-illuminant color correction 
model's output color space. An image captured by an electronic 
camera may have incorrect white point, due to incorrect estimation 
of illumination automatically made on the camera. For such a case, 
white point correction can be readily made off the camera after 
capturing the image, if the color correction model's output space is 
XYZ. If the output space is a nonlinear color space, white point 
correction is more complicated because the output data has to be 
once brought into a linear space to perform the white point 
correction. 

It can be assumed that an electronic camera's color 
transformation is a function of its white-balance setting, as 
described above In relation to llluminant dependency. No matter 
how correct or incorrect the on-camera estimation of illumination is, 
the camera's color transformation can be uniquely determined by 
referring to the camera's white-balance information, R/G and B/G for 
instance. Therefore, XYZ values that output from a multi-illuminant 
color correction model can well describe the original color's physical 
property, even if the input image's white point is incorrect. When a 
user determines that the white point is incorrect in an image taken 
by an electronic camera, the user can first output image data in XYZ 
from the multi-illuminant color correction model, then can re-estimate 
the image's illumination using a sophisticated method, which is too 
computationally expensive to be performed on-camera. The method 
can provide accuracy not only in white point, but also in color 
distributed around the re-balanced white point, because the re- 
balancing is applied to image data in corrected color space. A non- 
normalized linear color space can also be a space formed at an 
image sensor's output (raw image data). 



• 
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Color constancy 

It is known that many users prefer an electronic camera that 
reproduces color as if images were captured under one fixed 
standard illuminant such as D50 (specified by CIE, 5000K daylight), 
no matter what illuminant is actually used at the image capturing. 
This is referred to as "color constancy" color reproduction. By 
appropriately modifying the training data set of the above described 
multi-illuminant color correction model using either an MLP or a 
CANFIS neuro-fuzzy architecture, the above described "color- 
constancy" color correction is easily accomplished. 

It is to be understood that although an MLP is described as 
one example of a neural network which is applicable to the present 
invention, other types of neural networks such as a radial basis 
function network ("RBFN") or a modular neural network may also be 
utilized. 

It is to be further understood that each embodiment of the 
present invention may be implemented as software to be run on a 
computer or may be implemented as a function of image processing 
software run on a computer. In such an instance, image data files 
may be opened by the software, and any embodiment of the present 
invention may be applied to image data, such as a JPEG, GIF or BMP 
format file to allow color correction to be applied to such image 
data. 

It is additionally to be understood that the methods and 
apparatus of the present invention may be applied to characterize a 
component i*^ a color imaging device, such as an image sensor in an 
electronic camera. It is to be further understood that such 
characterization may be employed to correct for undesired qualities 
of such image sensor. 

Although only certain embodiments have been described in 
detail, those having ordinary skill in the art will certainly understand 
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that many modifications are possible without departing from the 
teachings hereof. All such modifications are intended to be 
encompassed within the following claims. 
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We claim: 

1 . A method of correcting color of a color image obtained 
by an electronic camera, comprising the steps of: 

determining, using a neural network, a correction to 
5 data representative of the color image based upon an 

estimated illuminant of the color image; and 

applying the correction to the data representative of the 
color image, wherein the illuminant comprises multiple sources 
of illumination. 

10 2. The method of claim 1, wherein the electronic camera 

captures at least one still image.. 

3. The method of claim 1 , wherein the electronic camera 
captures a succession of moving images. 

4. A method of correcting color of a color image obtained 
by an electronic camera, comprising the steps of: 

determining, using a multilayer perceptron model, a 
correction to data representative of the color image based 
upon an estimated illuminant of the color image; and 

applying the correction to the data representative of the 
color image, wherein the illuminant comprises multiple sources 
of illumination. 

5. The method of claim 4, wherein the electronic camera 
captures at least one still image. 

6. The method of claim 4, wherein the electronic camera 
25 captures a succession of moving images. 
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7. The method of claim 4, wherein the multilayer 
perceptron model is trained based upon a dogleg trust region 
implementation of a Levenberg-Marquardt type algorithm. 

8. The method of claim 4, further comprising the step of: 
5 outputting an output color space of the color corrected 

image as a space not normalized with chromaticity 
coordinates the sources of illumination. 



9. The method of claim 4, further comprising the step of: 
using training data of each neural network as a 

10 colorimetric value under a standard source of illumination. 

10. A method of correcting color of a color image obtained 
by an electronic camera, comprising the steps of: 

determining, using a coactive neuro-fuzzy inference 
system model, a correction to data representative of the color 
1 5 image based upon an estimated illuminant of the color image; 

and 

applying the correction to the data representative of the 
color image, wherein the illuminant comprises multiple sources 
of illumination. 



20 11. The method of claim 1 0, wherein the electronic camera 

captures at least one still image. 

12. The method of claim 10, wherein the electronic camera 
captures a succession of moving images. 
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cornprised of fuzzy membership functions computes a weighted sum 
of outputs of local expert multilayer perceptrons based upon an on- 
camera estimation of illumination at a time of color image capture. 



5 14. The method of claim 13, further comprising the step of: 

constructing fuzzy membership functions by applying a 
neural network nonlinear coordinate transformation to a white 
balance plane in order to characterize estimated illumination 
for the coactive neuro-fuzzy inference system model. 



10 15. The method of claim 10, further comprising the step of: 

training the coactive neuro-fuzzy inference system 
model by constructing fuzzy membership functions generated 
by applying a neural network nonlinear coordinate 
transformation to a white balance plane in order to 

15 characterize estimated illumination for the coactive neuro- 

fuzzy inference system model. 

1 6. The method of claim 1 3, further comprising the step of: 

training the coactive neuro-fuzzy inference system 
model by constructing fuzzy membership functions generated 

20 by applying a neural network nonlinear coordinate 

transformation to a white balance plane in order to 
characterize estimated illumination for the coactive neuro- 
fuzzy inference system model, wherein all parameters of fuzzy 
membership functions and local expert multilayer perceptrons 

25 are updated simultaneously. 
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17. The method of claim 13, further comprising the step of: 
training the coactive neuro-fuzzy inference system 

model by constructing fuzzy membership functions generated 
by applying a neural network nonlinear coordinate 
transformation to a white balance plane in order to 
characterize estimated illumination for the coactive neuro- 
fuzzy inference system model, wherein all parameters of fuzzy 
membership functions and local expert multilayer perceptrons 
are updated simultaneously in conjunction with a heuristic 
parameter updating rule. 

18. The method of claim 1 3, 1 4, 1 5, 1 6 or 1 7, wherein at 
least two of the fuzzy membership functions overlap. 

19. A method of correcting color of a color image obtained 
by an electronic camera, comprising the steps of: 



system with a switching unit, a correction to data 
representative of the color image based upon an estimated 
iliuminant of the color image. 

20. The method of claim 10, further comprising the steps 



finding a color conversion inverse map using separate 
neural networks associated with respective representative 
sources of illumination; and 

outputting an output color space of the color corrected 
image as a space not normalized with chromaticity 
coordinates the sources of illumination. 



determining, using a coactive neuro-fuzzy inference 
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21. The method of claim 10, further comprising the steps 

of: 

finding a color conversion inverse map using neural 
networks associated with respective representative sources of 
5 illumination; and 

outputting an output color space of the color corrected 
image as a space not normalized with chromaticity 
coordinates the sources of illumination. 

22. The method of claim 10, further comprising the steps 

finding a color conversion inverse map using separate 
neural networks associated with respective representative 
sources of illumination; and 

using training data of each neural network as a 
colorimetric value under a standard source of illumination. 

23. The method of claim 10, further comprising the steps 

finding a color conversion inverse map using neural 
networks associated with respective representative sources of 
illumination; and 

using training data of each neural network as a 
colorimetric value under a standard source of illumination. 

24. An apparatus for correcting color of a color image 
obtained by an electronic camera, comprising: 

25 a neural network for determining a correction to data 

representative of the color image based upon an estimated 
illuminant of the color image and for applying the correction to 
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the data representative of the color image, wherein the 
illuminant comprises multiple sources of illumination. 

25. The apparatus of claim 24, wherein the electronic 
camera captures at least one still image. 

5 26. The apparatus of claim 24, wherein the electronic 

camera captures a succession of moving images. 



27. An apparatus for correcting color of a color image 
obtained by an electronic camera, comprising: 

a multilayer perceptron model for determining a 
10 correction to data representative of the color image based 

upon an estimated illuminant of the color image, and for 
applying the correction to the data representative of the color 
image, wherein the illuminant comprises multiple sources of 
illumination. 



1 5 28. The apparatus of claim 27, wherein the electronic 

camera captures at least one still image. 

29. The apparatus of claim 27, wherein the electronic 
camera captures a succession of moving images. 

30. The apparatus of claim 27, wherein the multilayer 
20 perceptron model is trained based upon a dogleg trust region 

implementation of a Levenberg-Marquardt type algorithm. 
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31. The apparatus of claim 27, wherein the multilayer 
perpeptron model outputs an output color space of the color 
corrected image as a space not normalized with chromaticity 
coordinates the sources of illumination. 



5 32. The apparatus of claim 27, wherein the multilayer 

perceptron model uses training data of each neural network as a 
colorimetric value under a standard source of illumination. 



33. An apparatus for correcting color of a color image 
obtained by an electronic camera, comprising: 

10 a coactive neuro-fuzzy inference system model for 

determining a correction to data representative of the color 
image based upon an estimated illuminant of the color image, 
and for applying the correction to the data representative of 
the color image, wherein the illuminant comprises multiple 

15 sources of illumination. 



34. The apparatus of claim 33, wherein the electronic 
camera captures at least one still image. 

35. The apparatus of claim 33, wherein the electronic 
camera captures a succession of moving images. 

20 36. The apparatus of claim 33, wherein an integrating unit 

comprised of fi^zzy membership functions computes a weighted sum 
of outputs of local expert multilayer perceptrons based upon an on- 
camera estimation of illumination at a time of color image capture. 



# 



00/65847 



PCT/USOO/11504 



-44- 



37. The apparatus of claim 36, wherein fuzzy membership 
functions are constructed by applying a neural network nonlinear 
coordinate transformation to a white balance plane in order to 
characterize estimated illumination for the coactive neuro-fuzzy 
inference system model. 

38. The apparatus of claim 33, wherein the coactive neuro- 
fuzzy inference system model is trained by constructing fuzzy 
membership functions generated by applying a neural network 
nonlinear coordinate transformation to a white balance plane in order 
to characterize estimated illumination for the coactive neuro-fuzzy 
inference system model. 

39. The apparatus of claim 36, wherein the coactive neuro- 
fuzzy inference system model is trained by constructing fuzzy 
membership functions generated by applying a neural network 
nonlinear coordinate transformation to a white balance plane in order 
to characterize estimated illumination for the coactive neuro-fuzzy 
inference system model, wherein all parameters of fuzzy membership 
functions and local expert multilayer perceptrons are updated 
simultaneously. 

40. The apparatus of claim 36, wherein the coactive neuro- 
fuzzy inference system model is trained by constructing fuzzy 
membership functions generated by applying a neural network 
nonlinear coordinate transformation to a white balance plane in order 
to characterize estimated illumination for the coactive neuro-fuzzy 
inference system model, wherein all parameters of fuzzy membership 
functions and local expert multilayer perceptrons are updated 
simultaneously in conjunction with a heuristic parameter updating 
rule. 
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41. The apparatus of claim 36, 37, 38, 39 or 40, wherein 
at least two of the fuzzy membership functions overlap. 

42. An apparatus for correcting color of a color image 
obtained by an electronic camera, comprising: 

5 a coactive neuro-fuzzy inference system with a 

switching unit for determining a correction to data 
representative of the color image based upon an estimated 
illuminant of the color image. 



43. The apparatus of claim 33, wherein coactive neuro- 
1 0 fuzzy inference system model finds a color conversion inverse map 

using separate neural networks associated with respective 
representative sources of illumination, and outputs an output color 
space of the color corrected image as a space not normalized with 
chromaticity coordinates the sources of illumination. 



1 5 44. The apparatus of claim 33, wherein coactive neuro- 

fuzzy inference system model finds a color conversion inverse map 
using neural networks associated with respective representative 
sources of illumination, and outputs an output color space of the 
color corrected image as a space not normalized with chromaticity 

20 coordinates the sources of illumination. 



45. The apparatus of claim 33, wherein coactive neuro- 
fuzzy inference system model finds a color conversion inverse map 
using separate neural networks associated with respective 
representative sources of illumination, and uses training data of each 
25 neural network as a coiorimetric value under a standard source of 

illumination. 
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46. The apparatus of claim 33, wherein coactive neuro- 
fuzzy inference system model finds a color conversion inverse map 
using neural networks associated with respective representative 
sources of illumination, and uses training data of each neural 
network as a colorimetric value under a standard source of 
illumination. 

47. A recording medium having recorded thereon color 
corrected data of a color image obtained by an electronic camera, 
the recording medium being prepared by the steps of: 

determining, using a neural network, a correction to 
data representative of the color image based upon an 
estimated illumtnant of the color image; 

applying the correction to the data representative of the 
color image, wherein the illuminant comprises multiple sources 
of illumination; and 

recording on the recording medium data representative 
of the corrected data. 

48. The recording medium 47, wherein the electronic 
camera captures at least one still image. 

49. The recording medium of claim 47, wherein the 
electronic camera captures a succession of moving images. 

50. A recording medium having recorded thereon color 
corrected data of a color image obtained by an electronic camera, 
the recording medium being prepared by the steps of: 

determining, using a multilayer perceptron model, a 
correction to data representative of the color image based 
upon an estimated illuminant of the color image; 



• 
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applying the correction to the data representative of the 
color image, wherein the illuminant comprises multiple sources 
of illumination; and 

recording on the recording medium data representative 
5 of the corrected data. 



51 . The recording medium of claim 50, wherein the 
electronic camera captures at least one still image. 

52. The recording medium of claim 50, wherein the 
electronic camera captures a succession of moving images. 

10 53. The recording medium of claim 50, wherein the 

multilayer perceptron model is trained based upon a dogleg trust 
region implementation of a Levenberg-Marquardt type algorithm. 



54. The recording medium of claim 54, further comprising 
the step of: 

1 5 outputting an output color space of the color corrected 

image as a space not normalized with chromaticity 
coordinates the sources of illumination. 



55. The recording medium of claim 50, further comprising 
the step of: 

20 using training data of each neural network as a 

colorimetric value under a standard source of illumination. 



m 
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56. A recording medium having recorded thereon color 
corrected data of a color image obtained by an electronic camera, 
the recording medium being prepared by the steps of: 

determining, using a coactive neuro-fuzzy inference 
system model, a correction to data representative of the color 
image based upon an estimated illuminant of the color image; 

applying the correction to the data representative of the 
color image, wherein the illuminant comprises multiple sources 
of illumination; and 

recording on the recording medium data representative 
of the corrected data. 

57. The recording medium of claim 56, wherein the 
electronic camera captures at least one still image. 

58. The recording medium of claim 56, wherein the 
electronic camera captures a succession of moving images. 

59. The recording medium of claim 56, wherein an 
integrating unit comprised of fuzzy membership functions computes 
a weighted sum of outputs of local expert multilayer perceptrons 
based upon an on-camera estimation of illumination at a time of 
color image capture. 

60. The recording medium of claim 59, further comprising 
the step of: 

constructing fuzzy membership functions by applying a 
neural network nonlinear coordinate transformation to a white 
balance plane in order to characterize estimated illumination 
for the coactive neuro-fuzzy inference system model. 
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61. The recording medium of claim 56, further comprising 
the -Step of: 

training the coactive neuro-fuzzy inference system 
model by constructing fuzzy membership functions generated 
5 by applying a neural network nonlinear coordinate 

transformation to a white balance plane in order to 
characterize estimated illumination for the coactive neuro- 
fuzzy inference system model. 

62. The recording medium of claim 59, further comprising 
the step of: 

training the coactive neuro-fuzzy inference system 
model by constructing fuzzy membership functions generated 
by applying a neural network nonlinear coordinate 
transformation to a white balance plane in order to 
characterize estimated illumination for the coactive neuro- 
fuzzy inference system model, wherein all parameters of fuzzy 
membership functions and local expert multilayer perceptrons 
are updated simultaneously. 

63. The recording medium of claim 59, further comprising 
20 the step of: 

training the coactive neuro-fuzzy inference system 
model by constructing fuzzy membership functions generated 
by applying a neural network nonlinear coordinate 
transformation to a white balance plane in order to 
25 characterize estimated illumination for the coactive neuro- 

fuzzy inference system model, wherein ail parameters of fuzzy 
membership functions and local expert multilayer perceptrons 
are updated simultaneously in conjunction with a heuristic 
parameter updating rule. 
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64. The recording medium of claim 59, 60, 61, 62 or 63, 
wherein at least two of the fuzzy membership functions overlap. 

65. A recording medium having recorded thereon color 
corrected data of a color image obtained by an electronic camera, 

5 the recording medium being prepared by the steps of: 

determining, using a coactive neuro-fuzzy inference 
system with a switching unit, a correction to data 
representative of the color image based upon an estimated 
illuminant of the color image; and 
10 recording on the recording medium data representative 

of the corrected data. 

66. The recording medium of claim 56, further comprising 
the steps of: 

finding a color conversion inverse map using separate 
15 neural networks associated with respective representative 

sources of illumination; and 

outputting an output color space of the color corrected 
image as a space not normalized with chromaticity 
coordinates the sources of illumination. 



67. The recording medium of claim 56, further comprising 
the steps of: 

finding a color conversion inverse map using neural 
networks associated with respective representative sour res of 
illumination; and 

outputting an output color space of the color corrected 
image as a space not normalized with chromaticity 
coordinates the sources of illumination. 
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68. The recording medium of claim 56, further comprising 
the steps of: 

finding a color conversion inverse map using separate 
neural networks associated with respective representative 
5 sources of illumination; and 

using training data of each neural network as a 
colorimetric value under a standard source of illumination. 

69. The recording medium of claim 56, further comprising 
the steps of: 

10 finding a color conversion inverse map using neural 

networks associated with respective representative sources of 
illumination; and 

using training data of each neural network as a 
colorimetric value under a standard source of illumination. 

70. A method of transmitting color corrected data of a color 
image obtained by an electronic camera, comprising the steps of: 

determining, using a neural network, a correction to 
data representative of the color image based upon an 
estimated illuminant of the color image; 

applying the correction to the data representative of the 
color image, wherein the illuminant comprises multiple sources 
of illumination; and 

transmitting data representative of the corrected data. 

71. The method of claim 70, wherein the electronic camera 
25 captures at least one still image. 

72. The method of claim 70, wherein the electronic camera 
captures a succession of moving images. 
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73. A method of transmitting color corrected data of a color 
image obtained by an electronic camera, comprising the steps of: 
determining, using a multilayer perceptron model, a 
correction to data representative of the color image based 
5 upon an estimated illuminant of the color image; 

applying the correction to the data representative of the 
color image, wherein the illuminant comprises multiple sources 
of illumination; and 

transmitting data representative of the corrected data. 

10 74. The method of claim 73, wherein the electronic camera 

captures at least one still image. 

75. The method of claim 73, wherein the electronic camera 
captures a succession of moving images. 

76. The method of claim 73, wherein the multilayer 
1 5 perceptron model is trained based upon a dogleg trust region 

implementation of a Levenberg-Marquardt type algorithm. 

77. The method of claim 73, further comprising the step of: 
outputting an output color space of the color corrected 

image as a space not normalized with chromaticity 
20 coordinates the sources of illumination. 

78. The method of claim 79, further comprising the step of: 
using training data of each neural network as a 

colorimetric value under a standard source of illumination. 
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79. A method of transmitting color corrected data of a color 
image obtained by an electronic camera, comprising the steps of: 
determining, using a coactive neuro-fuzzy inference 
system model, a correction to data representative of the color 
5 image based upon an estimated illuminant of the color images- 

applying the correction to the data representative of the 
color image, wherein the illuminant comprises multiple sources 
of illumination; and 

transmitting data representative of the corrected data. 

10 80- The method of claim 79, wherein the electronic camera 

captures at least one still image. 

81. The method of claim 79, wherein the electronic camera 
captures a succession of moving images. 

82. The method of claim 79, wherein an integrating unit 

1 5 comprised of fuzzy membership functions computes a weighted sum 

of outputs of local expert multilayer perceptrons based upon an on- 
camera estimation of illumination at a time of color image capture. 

83. The method of claim 82, further comprising the step of: 
constructing fuzzy membership functions by applying a 

20 neural network nonlinear coordinate transformation to a white 

balance plane in order to characterize estimated illumination 
for the coactive neuro-fuzzy inference system model. 

84. The method of claim 79, further comprising the step of: 
training the coactive neuro-fuzzy inference system 

25 model by constructing fuzzy membership functions generated 

by applying a neural network nonlinear coordinate 
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transformation to a white balance plane in order to 
characterize estimated illumination for the coactive neuro- 
fuzzy inference system model. 

85. The method of claim 82, further comprising the step of: 
5 training the coactive neuro-fuzzy inference system 

model by constructing fuzzy membership functions generated 
by applying a neural network nonlinear coordinate 
transformation to a white balance plane in order to 
characterize estimated illumination for the coactive neuro- 
10 fuzzy inference system model, wherein all parameters of fuzzy 

membership functions and local expert multilayer perceptrons 
are updated simultaneously. 

86. The method of claim 82, further comprising the step of: 
training the coactive neuro-fuzzy inference system 

1 5 model by constructing fuzzy membership functions generated 

by applying a neural network nonlinear coordinate 
transformation to a white balance plane in order to 
characterize estimated illumination for the coactive neuro- 
fuzzy inference system model, wherein all parameters of fuzzy 

20 membership functions and local expert multilayer perceptrons 

are updated simultaneously in conjunction with a heuristic 
parameter updating rule. 



87. The method of claim 82, 83, 84, 85 or 86 wherein at 
least two of the fuzzy membership functions overlap. 
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88. A method of transmitting color corrected data of a color 
image obtained by an electronic camera, comprising the steps of: 
determining, using a coactive neuro-fuzzy inference 
system with a switching unit, a correction to data 
representative of the color image based upon an estimated 
illuminant of the color image; and 

transmitting data representative of the corrected data. 



89. The method of claim 79, further comprising the steps 

finding a color conversion inverse map using separate 
neural networks associated with respective representative 
sources of illumination; and 

outputting an output color space of the color corrected 
image as a space not normalized with chromaticity 
coordinates the sources of illumination. 



90. The method of claim 79, further comprising the steps 

finding a color conversion inverse map using neural 
networks associated with respective representative sources of 
illumination; and 

outputting an output color space of the color corrected 
image as a space not normalized with chromaticity 
coordinates the sources of illumination. 



91 . The method of claim 79, further comprising the steps 

finding a color conversion inverse map using separate 
neural networks associated with respective representative 
sources of illumination; and 
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15 



using training data of each neural network as a 
colorimetric value under a standard source of illumination. 

92. The method of claim 79, further comprising the steps 

of: 

finding a color conversion inverse map using neural 
networks associated with respective representative sources of 
illumination; and 

using training data of each neural network as a 
colorimetric value under a standard source of illumination. 

93. The method of claim 1 , 4 or 10, wherein the data 
representative of the color image includes information regarding the 
illuminant. 

94. The apparatus of claim 24, 27 or 33, wherein the data 
representative of the color image includes information regarding the 
illuminant. 

95. A method of recording image data obtained by an 
electronic camera, comprising the steps of: 

capturing a color image and generating data representative of 
the image; 

estimating an illuminant for the captured color image and 
generating data representative of the estimated illuminant; and 

recording the data representative of the image with the data 
representative of the estimated illuminant. 
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96. A method of transmitting image data obtained by an 
electronic camera, comprising the steps of: 

capturing a color image and generating data representative of 
the image; 

estimating an illuminant for the captured color image and 
generating data representative of the estimated illuminant; and 

transmitting the data representative of the image with the 
data representative of the estimated illuminant. 
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AMENDED CLAIMS 

[received by the International Bureau on 18 September 2000 (18.09.00); 
original claim 20-23, 43-46, 66-69 and 89-92 amended; 
remaining claims unchanged (23 pages)] 

' 1 . A method of correcting color of a color image obtained 
by an electronic camera, comprising the steps of: 

determining, using a neural network, a correction to 
data representative of the color image based upon an 
estimated iliuminant of the color image; and 

applying the correction to the data representative of the 
color image, wherein the iliuminant comprises multiple sources 
of illumination. 

2. The method of claim 1 , wherein the electronic camera 
captures at least one still image. 

3. The method of claim 1, wherein the electronic camera 
captures a succession of moving images. 

4. A method of correcting color of a color image obtained 
by an electronic camera, comprising the steps of: 

determining, using a multilayer perceptron model, a 
correction to data representative of the color image based 
upon an estimated iliuminant of the color image; and 

applying the correction to the data representative of the 
color image, wherein the iliuminant comprises multiple sources 
of illumination. 

5. The method of claim 4, wherein the electronic camera 
captures at least one still image. 
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6. The method of claim 4, wherein the electronic camera 
captures a succession of moving images. 

7. The method of claim 4, wherein the multilayer 

5 perceptron model is trained based upon a dogleg trust region 
implementation of a Levenberg-Marquardt type algorithm. 

8. The method of claim 4, further comprising the step of: 
outputting an output color space of the color corrected 

10 image as a space not normalized with chromaticity 

coordinates the sources of illumination. 

9. The method of claim 4, further comprising the step of: 
using training data of each neural network as a 

15 colorimetric value under a standard source of illumination. 



10. A method of correcting color of a color image obtained 
by an electronic camera, comprising the steps of: 

determining, using a coactive neuro-fuzzy inference 
20 system model, a correction to data representative of the color 

image based upon an estimated illuminant of the color image; 
and 

applying the correction to the data representative of the 
color image, wherein the illuminant comprises multiple sources 
25 of illumii^ation. 



1 1 . The method of claim 10, wherein the electronic camera 
captures at least one still image. 
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12. The method of claim 10, wherein the electronic camera 
captures a succession of- moving images. 

13. The method of claim 10, wherein an integrating unit 

5 comprised of fuzzy membership functions computes a weighted sum 
of outputs of local expert multilayer perceptrons based upon an on- 
camera estimation of illumination at a time of color image capture. 

14. The method of claim 13, further comprising the step of: 
10 constructing fuzzy membership functions by applying a 

neural network nonlinear coordinate transformation to a white 
balance plane in order to characterize estimated illumination 
for the coactive neuro-fuzzy inference system model. 

15 15. The method of claim 10, further comprising the step of: 

training the coactive neuro-fuzzy inference system 
model by constructing fuzzy membership functions generated 
by applying a neural network nonlinear coordinate 
transformation to a white balance plane in order to 

20 characterize estimated illumination for the coactive neuro- 

fuzzy inference system model. 

16. The method of claim 13, further comprising the step of: 
training the coactive neuro-fuzzy inference system 
25 model by constructing fuzzy membership functions generated 

by applying a neural network nonlinear coordinate 
transformation to a white balance plane in order to 
characterize estimated illumination for the coactive neuro- 
fuzzy inference system model, wherein all parameters of fuzzy 
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membership functions and local expert multilayer perceptrons 
are updated simultaneously. 

17. The method of claim 1 3, further comprising the step of: 
5 training the coactive neuro-fuzzy inference system 

model by constructing fuzzy membership functions generated 
by applying a neural network nonlinear coordinate 
transformation to a white balance plane in order to 
characterize estimated illumination for the coactive neuro- 
10 fuzzy inference system model, wherein all parameters of fuzzy 

membership functions and local expert multilayer perceptrons 
are updated simultaneously in conjunction with a heuristic 
parameter updating rule. 

15 18. The method of claim 13, 14, 15, 16 or 17, wherein at 

least two of the fuzzy membership functions overlap. 

19. A method of correcting color of a color image obtained 
by an electronic camera, comprising the steps of: 
20 determining, using a coactive neuro-fuzzy inference 

system with a switching unit, a correction to data 
representative of the color image based upon an estimated 
illuminant of the color image. 

25 20. The method of claim 10, further comprising the steps 

of: 

finding a color conversion inverse map using a plurality 
of neural networks associated with respective representative 
sources of illumination; 
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using a respective subset of training data to 
independently train each neural network in a component-wise 
manner, wherein each of the neural networks is a local expert 
neural network; and 
5 outputting an output color space of the color corrected 

image as a space with chromaticity coordinates of the sources 
of illumination. 

21 . The method of claim 10, further comprising the steps 

10 of: 

finding a color conversion inverse map using a plurality 
of neural networks associated with respective representative 
sources of illumination; 

using an entire training data set to simultaneously train 
15 all of the neural networks in a system-wise manner, wherein 
the neural networks are local expert neural networks; and 

outputting an output color space of the color corrected 
image as a space with chromaticity coordinates of the sources 
of illumination. 

20 

22. The method of claim 10, further comprising the steps 

of: 

finding a color conversion inverse map using neural 
networks associated with respective representative sources of 
25 illumination; and 

using a respective subset of training data to 
independently train each neural network in a component-wise 
manner, wherein each of the neural networks is a local expert 
neural network. 
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23. The method of claim 10, further comprising the steps 

of: 

finding a color conversion inverse map using neural 
5 networks associated with respective representative sources of 
illumination; and 

using an entire training data set to simultaneously train 
all of the neural networks in a system-wise manner, wherein 
the neural networks are local expert neural networks. 

10 

24. An apparatus for correcting color of a color image 
obtained by an electronic camera, comprising: 

a neural network for determining a correction to data 
representative of the color image based upon an estimated 
15 illuminant of the color image and for applying the correction to 

the data representative of the color image, wherein the 
illuminant comprises multiple sources of illumination. 

25. The apparatus of claim 24, wherein the electronic 
20 camera captures at least one still image. 

26. The apparatus of claim 24, wherein the electronic 
camera captures a succession of moving images. 

25 27. An apparatus for correcting color of a color image 

obtained by an electronic camera, comprising: 

a multilayer perceptron model for determining a 
correction to data representative of the color image based 
upon an estimated illuminant of the color image, and for 
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applying the correction to the data representative of the color 
image, wherein the illunninant comprises multiple sources of 
illumination. 

5 28. The apparatus of claim 27, wherein the electronic 

camera captures at least one still image. 

29. The apparatus of claim 27, wherein the electronic 
camera captures a succession of moving images. 

10 

30. The apparatus of claim 27, wherein the multilayer 
perceptron model is trained based upon a dogleg trust region 
implementation of a Levenberg-Marquardt type algorithm. 

15 31. The apparatus of claim 27, wherein the multilayer 

perceptron model outputs an output color space of the color 
corrected image as a space not normalized with chromaticity 
coordinates the sources of illumination. 

20 32. The apparatus of claim 27, wherein the multilayer 

perceptron model uses training data of each neural network as a 
colorimetric value under a stanuard source of illumination. 

33. An apparatus for correcting color of a color image 
25 obtained by an electronic camera, comprising: 

a coactive neuro-fuzzy inference system model for 
determining a correction to data representative of the color 
image based upon an estimated illuminant of the color image, 
and for applying the correction to the data representative of 
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the color image, wherein the illuminant comprises multiple 
sources of illumination. 

34. The apparatus of claim 33, wherein the electronic 
5 camera captures at least one still image. 

35. The apparatus of claim 33, wherein the electronic 
camera captures a succession of moving images. 

10 36. The apparatus of claim 33, wherein an integrating unit 

comprised of fuzzy membership functions computes a weighted sum 
of outputs of local expert multilayer perceptrons based upon an on- 
camera estimation of illumination at a time of color image capture. 

15 37. The apparatus of claim 36, wherein fuzzy membership 

functions are constructed by applying a neural network nonlinear 
coordinate transformation to a white balance plane in order to 
characterize estimated illumination for the coactive neuro-fuzzy 
inference system model. 

20 

38. The apparatus of claim 33, wherein the coactive neuro- 
fuzzy inference system model is trained by constructing fuzzy 
membership functions generated by applying a neural network 
nonlinear coordinate transformation to a white balance plane in order 

25 to characterize estimated illumination for the coactive neuro-fuzzy 
inference system model. 

39. The apparatus of claim 36, wherein the coactive neuro- 
fuzzy inference system model is trained by constructing fuzzy 
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10 



15 



20 



25 



membership functions generated by applying a neural network 
nonlinear coordinate transformation to a white balance plane in order 
to characterize estimated illumination for the coactive neuro-fuzzy 
inference system model, wherein all parameters of fuzzy membership 
functions and local expert multilayer perceptrons are updated 
simultaneously. 

40. The apparatus of claim 36, wherein the coactive neuro- 
fuzzy inference system model is trained by constructing fuzzy 
membership functions generated by applying a neural network 
nonlinear coordinate transformation to a white balance plane in order 
to characterize estimated illumination for the coactive neuro-fuzzy 
inference system model, wherein all parameters of fuzzy membership 
functions and local expert multilayer perceptrons are updated 
simultaneously in conjunction with a heuristic parameter updating 
rule. 

41. The apparatus of claim 36, 37, 38, 39 or 40, wherein 
at least two of the fuzzy membership functions overlap. 

42. An apparatus for correcting color of a color image 
obtained by an electronic camera, comprising: 

a coactive neuro-fuzzy inference system with a 
switching unit for determining a correction to data 
representative of the color image based upon an estimated 
illuminant of the color image. 

43. The apparatus of claim 33, wherein the coactive neuro- 
fuzzy inference system model finds a color conversion inverse map 
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using a plurality of neural networks associated with respective 
representative sources of illumination, uses a respective subset of 
training data to independently Lain each neural network in a 
component-wise manner, wherein each of the neural networks is a 
5 local expert neural network, and outputs an output color space of 
the color corrected image as a space with chromaticity coordinates 
of the sources of illumination. 



44. The apparatus of claim 33, wherein the coactive neuro- 
10 fuzzy inference system model finds a color conversion inverse map 

using a plurality of neural networks associated with respective 
representative sources of illumination, uses an entire training data 
set to simultaneously train all of the neural networks in a system- 
wise manner, wherein the neural networks are local expert neural 
15 networks, and outputs an output color space of the color corrected 
image as a space with chromaticity coordinates of the sources of 
illumination. 

45. The apparatus of claim 33, wherein the coactive neuro- 
20 fuzzy inference system model finds a color conversion inverse map 

using neural networks associated with respective representative 
sources of illumination, and use<=; a respective subset of training data 
to independently train each neural network in a component-wise 
manner, wherein each of the neural networks is a local expert neural 
25 network. 



46. The apparatus of claim 33, wherein the coactive neuro- 
fuzzy inference system model finds a color conversion inverse map 
using neural networks associated with respective representative 
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sources of illumination, and uses an entire training data set to 
simultaneously train all of the neural networks in a system-wise 
manner, wherein the neural networks are local expert neural 
networks. 

5 

47. A recording medium having recorded thereon color 
corrected data of a color image obtained by an electronic camera, 
the recording medium being prepared by the steps of: 

determining, using a neural network, a correction to 
10 data representative of the color image based upon an 

estimated illuminant of the color image; 

applying the correction to the data representative of the 
color image, wherein the illuminant comprises multiple sources 
of illumination; and 
15 recording on the recording medium data representative 

of the corrected data. 

48. The recording medium 47, wherein the electronic 
camera captures at least one still image. 

20 

49. The recording medium of claim 47, wherein the 
electronic camera captures a succession of moving images. 

50. A recording medium having recorded thereon color 
25 corrected data of a color image obtained by an electronic camera, 

the recording medium being preoared by the steps of: 

determining, using a multilayer perceptron model, a 
correction to data representative of the color image based 
upon an estimated illuminant of the color image; 
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applying the correction to the data representative of the 
color image, wherein the illuminant comprises multiple sources 
of illumination; and 

recording on the recording medium data representative 
5 of the corrected data. 

51 . The recording medium of claim 50, wherein the 
electronic camera captures at least one still image. 

10 52. The recording medium of claim 50, wherein the 

electronic camera captures a succession of moving images. 

53. The recording medium of claim 50, wherein the 
multilayer perceptron model is trained based upon a dogleg trust 

15 region implementation of a Levenberg-Marquardt type algorithm. 

54. The recording medium of claim 54, further comprising 
the step of: 

outputting an output color space of the color corrected 
20 image as a space not normalized with chromaticity 

coordinates the sources of illumination. 

55. The recording medium of claim 50, further comprising 
the step of: 

25 using training data of each neural network as a 

colorimetric value under a standard source of illumination. 
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56. A recording medium having recorded thereon color 
corrected data of a cole image obtained by an electronic camera, 
the recording medium being prepared by the steps of: 

determining, using a coactive neuro-fuzzy inference 
5 system model, a correction to data representative of the color 

image based upon an estimated illuminant of the color image; 

applying the correction to the data representative of the 
color image, wherein the illuminant comprises multiple sources 
of illumination; and 

10 recording on the recording medium data representative 

of the corrected data. 



57. The recording medium of claim 56, wherein the 
electronic camera captures at least one still image. 

15 

58. The recording medium of claim 56, wherein the 
electronic camera captures a succession of moving images. 

59. The recording medium of claim 56, wherein an 

20 integrating unit comprised of fuzzy membership functions computes 
a weighted sum of outputs of local expert multilayer perceptrons 
based upon an on-camera estimation of illumination at a time of 
color image capture. 

25 60. The recording medium of claim 59, further comprising 

the step of: 

constructing fuzzy membership functions by applying a 
neural network nonlinear coordinate transformation to a white 
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balance plane in order to characterize estimated illunnination 
for the coactive neuro-fuzzy inference system model. 

61. The recording medium of claim 56, further comprising 
the step of: 

training the coactive neuro-fuzzy inference system 
model by constructing fuzzy membership functions generated 
by applying a neural network nonlinear coordinate 
transformation to a white balance plane in order to 
characterize estimated illumination for the coactive neuro- 
fuzzy inference system model. 

62. The recording medium of claim 59, further comprising 
the step of: 

training the coactive neuro-fuzzy inference system 
model by constructing fuzzy membership functions generated 
by applying a neural network nonlinear coordinate 
transformation to a white balance plane in order to 
characterize estimated illumination for the coactive neuro- 
fuzzy inference system model, wherein all parameters of fuzzy 
membership functions and local expert multilayer perceptrons 
are updated simultaneously. 

63. The recording medium of claim 59, further comprising 
25 the step of: 

training the coactive neuro-fuzzy inference system 
model by constructing fuzzy membership functions generated 
by applying a neural network nonlinear coordinate 
transformation to a white balance plane in order to 
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characterize estimated illumination for the coactive neuro- 
fuzzy inference system model, wherein all parameters of fuzzy 
membership functions and local expert multilayer perceptrons 
are updated simultaneously in conjunction with a heuristic 
5 parameter updating rule. 

64. The recording medium of claim 59, 60, 61, 62 or 63, 
wherein at least two of the fuzzy membership functions overlap. 

10 65. A recording medium having recorded thereon color 

corrected data of a color image obtained by an electronic camera, 
the recording medium being prepared by the steps of: 

determining, using a coactive neuro-fuzzy inference 
system with a switching unit, a correction to data 
15 representative of the color image based upon an estimated 

illuminant of the color image; and 

recording on the recording medium data representative 
of the corrected data. 

20 66. The recording medium of claim 56, further comprising 

the steps of: 

finding a color conv'=*rsion inverse map using a plurality 
of neural networks associated with respective representative 
sources of illumination; 
25 using a respective subset of training data to 

independently train each neural network in a component-wise 
manner, wherein each of the neural networks is a local expert 
neural network; and 

outputting an output color space of the color corrected 
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image as a space with chromaticity coordinates of the sources 
of illumination. 

67. The recording medium of claim 56, further comprising 
5 the steps of: 

finding a color conversion inverse map using a plurality 
of neural networks associated with respective representative 
sources of illumination; 

using an entire training data set to simultaneously train 
10 all of the neural networks in a system-wise manner, wherein 
the neural networks are local expert neural networks; and 

outputting an output color space of the color corrected 
image as a space with chromaticity coordinates of the sources 
of illumination. 

15 

68. The recording medium of claim 56, further comprising 
the steps of: 

finding a color conversion inverse map using neural 
networks associated with respective representative sources of 
20 illumination; and 

using a respective subset of training data to 
independently train each neural network in a component-wise 
manner, wherein each of the neural networks is a local expert 
neural network. 

25 

69. The recording medium of claim 56, further comprising 
the steps of: 

finding a color conversion inverse map using neural 
networks associated with respective representative sources of 
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illumination; and 

using an entire training data set to simultaneously train 
all of the neural networks in a system-wise manner, wherein 
the neural networks are local expert neural networks. 

5 

70. A method of transmitting color corrected data of a color 
image obtained by an electronic camera, comprising the steps of: 

determining, using a neural network, a correction to 
data representative of the color image based upon an 
10 estimated illuminant of the color image; 

applying the correction to the data representative of the 
color image, wherein the illuminant comprises multiple sources 
of illumination; and 

transmitting data representative of the corrected data. 

15 

71 . The method of claim 70, wherein the electronic camera 
captures at least one still image. 

72. The method of claim 70, wherein the electronic camera 
20 captures a succession of moving images. 

73. A method of transmitting color corrected data of a color 
image obtained by an electronic camera, comprising the steps of: 

determining, using a multilayer perceptron model, a 
25 correction to data representative of the color image based 

upon an estimated illuminant of the color image; 

applying the correction to the data representative of the 
color image, wherein the illuminant comprises multiple sources 
of illumination; and 
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transmitting data representative of the corrected data. 

74. The method of claim 73, wherein the electronic camera 
captures at least one still image. 

5 

75. The method of claim 72, wherein the electronic camera 
captures a succession of moving images. 

76. The method of claim 73, wherein the multilayer 
10 perceptron model is trained based upon a dogleg trust region 

implementation of a Levenberg-Marquardt type algorithm. 

77. The method of claim 73, further comprising the step of: 
outputting an output color space of the color corrected 

15 image as a space not normalized with chromaticity 

coordinates the sources of illumination. 

78. The method of claim 79, further comprising the step of: 
using training data of each neural network as a 

20 colorimetric value under a standard source of illumination. 

79. A method of transmitting color corrected data of a color 
image obtained by an electronic camera, comprising the steps of: 

determining, using a coactive neuro-fuzzy inference 
25 system model, a correction to data representative of the color 

image based upon an estimated illuminant of the color image; 

applying the correction to the data representative of the 
color image, wherein the illuminant comprises multiple sources 
of illumination; and 
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transmitting data representative of the corrected data. 

80. The method of claim 79, wherein the electronic camera 
captures at least one still image. 

81. The method of claim 79, wherein the electronic camera 
captures a succession of moving images. 

82. The method of claim 79, wherein an integrating unit 
comprised of fuzzy membership functions computes a weighted sum 
of outputs of local expert multilayer perceptrons based upon an on- 
camera estimation of illumination at a time of color image capture. 



83. The method of claim 82, further comprising the step of: 
15 constructing fuzzy membership functions by applying a 

neural network nonlinear coordinate transformation to a white 
balance plane in order to characterize estimated illumination 
for the coactive neuro-fuzzy inference system model. 

20 84. The method of claim 79, further comprising the step of: 

training the coactive neuro-fuzzy inference system 
model by constructing fuzry membership functions generated 
by applying a neural network nonlinear coordinate 
transformation to a white balance plane in order to 

25 characterize estimated illumination for the coactive neuro- 

fuzzy inference system model. 

85. The method of claim 82, further comprising the step of: 
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training the coactive neuro-fuzzy inference system 
model by constructing fuzzy membership functions generated 
by applying a neural network nonlinear coordinate 
transformation to a white balance plane in order to 
characterize estimated illumination for the coactive neuro- 
fuzzy inference system model, wherein all parameters of fuzzy 
membership functions and local expert multilayer perceptrons 
are updated simultaneously. 

86. The method of claim 82, further comprising the step of: 

training the coactive neuro-fuzzy inference system 
model by constructing fuzzy membership functions generated 
by applying a neural network nonlinear coordinate 
transformation to a white balance plane in order to 
characterize estimated illumination for the coactive neuro- 
fuzzy inference system model, wherein all parameters of fuzzy 
membership functions and local expert multilayer perceptrons 
are updated simultaneously in conjunction with a heuristic 
parameter updating rule. 

87. The method of claim 82, 83, 84, 85 or 86 wherein at 
least two of the fuzzy membership functions overlap. 

88. A method of transmitting color corrected data of a color 
image obtained by an electronic camera, comprising the steps of: 

determining, using a coactive neuro-fuzzy inference 
system with a switching unit, a correction to data 
representative of the color image based upon an estimated 
illuminant of the color image; and 
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transmitting data representative of the corrected data. 

89. The method of claim 79, further comprising the steps 

of: 

5 finding a color conversion inverse map using a plurality 

of neural networks associated with respective representative 
sources of illumination; 

using a respective subset of training data to 
independently train each neural network in a component-wise 
10 manner, wherein each of the neural networks is a local expert 
neural network; and 

outputting an output color space of the color corrected 
image as a space with chromaticity coordinates of the sources 
of illumination, 

15 

90. The method of claim 79, further comprising the steps 

of: 

finding a color conversion inverse map using a plurality 
of neural networks associated with respective representative 
20 sources of illumination; 

using an entire training data set to simultaneously train 
all of the neural networks in a system-wise manner, wherein 
the neural networks are local expert neural networks; and 

outputting an output color space of the color corrected 
25 image as a space with chromaticity coordinates of the sources 
of illumination. 

91 . The method of claim 79, further comprising the steps 

of: 
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finding a color conversion inverse map using neural 
networks associated with respective representative sources of 
illumination; and 

using a respective subset of training data to 
5 independently train each neural network in a component-wise 
manner, wherein each of the neural networks is a local expert 
neural network. 

92. The method of claim 79, further comprising the steps 

10 of: 

finding a color conversion inverse map using neural 
networks associated with respective representative sources of 
illumination; and 

using an entire training data set to simultaneously train 
15 all of the neural networks in a system-wise manner, wherein 
the neural networks are local expert neural networks. 



93. The method of claim 1 , 4 or 10, wherein the data 
representative of the color image includes information regarding the 

20 illuminant. 

94. The apparatus of claim 24, 27 or 33, wherein the data 
representative of the color image includes information regarding the 
illuminant. 

25 

95. A method of recording image data obtained by an 
electronic camera, comprising the steps of: 

capturing a color image and generating data representative of 
the image; 
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estimating an illuminant for the captured color image and 
generating data representative of the estimated illuminant; and 

recording the data representative of the image with the data 
representative of the estimated illuminant. 

5 

96. A method of transmitting image data obtained by an 
electronic camera, comprising the steps of: 

capturing a color image and generating data representative of 
the image; 

10 estimating an illuminant for the captured color image and 

generating data representative of the estimated illuminant; and 

transmitting the data representative of the image with the 
data representative of the estimated illuminant. 
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